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Preface 


Preface 

About  This  Document 

This  report  documents  version  3.0  of  the  Software  Capability  Evaluation  (SCE)  Method.  This 
document  incorporates  Capability  Maturity  ModelSM1  (CMMsm)  vl.1  [Paulk  93a]  and  the  key 
practices  of  CMM  vl.1  [Paulk  93b]  into  the  SCE  Method.  SCE  V3.0  is  a  CAF-compliant 
method.2  The  primary  focus  of  this  report  is  on  what  is  done;  less  attention  is  given  to  how  it 
is  done.  SCE  Evaluator  training  provides  this  how-to  information. 

Some  of  the  objectives  for  the  SCE  Method  are  that  it  should  be  reliable,  repeatable,  trainable, 
consistent,  and  closely  aligned  with  the  CMM-Based  Appraisal  for  Internal  Process 
Improvement  (CBA IPI).  This  document  is  part  of  ongoing  efforts  to  meet  those  objectives  and 
to  improve  the  method. 

Goals  of  This  Document 

The  method  description  is  aimed  at  a  wide  audience.  After  reading  this  document,  readers 
should 

•  understand  how  SCE  fits  into  the  larger  context  of  software  acquisition, 
development,  and  process  improvement 

•  understand  the  fundamental  concepts  of  SCE 

•  know  what  the  activities  performed  during  an  SCE  are 

•  understand  the  objectives  of  an  SCE 

•  understand  the  relationship  with  other  CMM-based  diagnostic  tools 

Intended  Audience 

The  report  is  intended  to  help  managers  understand  the  SCE  Method,  help  sponsors  decide 
on  the  appropriateness  of  using  SCE  to  evaluate  a  target  organization,  and  enable  potential 
recipients  of  an  SCE  to  determine  if  an  internally  administered  SCE  would  point  out  areas  of 
risk  that  merit  immediate  improvement  to  compete  more  effectively.  The  audience  also 
includes  all  attendees  of  SCE  v3.0  training  and  anyone  who  wants  to  learn  about  the  SCE 
method. 


1-  Capability  Maturity  Model  and  CMM  are  service  marks  of  Carnegie  Mellon  University. 

2-  The  CMM  Appraisal  Framework  (CAF)  documents  appraisal  requirements  for  CMM-based  appraisal  methods. 
A  CAF  compliant  method  must  implement  all  CAF  requirements.  The  CAF  was  created  to  help  address  com¬ 
munity  concerns  for  achieving  better  alignment  between  SEI  CMM-based  appraisal  methods. 


CMU/SEI-96-TR-002 


©  1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


IX 


Preface 


April  1996 


It  is  assumed  that  the  reader  has  some  knowledge  of  the  SEI  Capability  Maturity  Model  (CMM) 
[Paulk  93a]  and  the  associated  document,  Key  Practices  of  the  Capability  Maturity  Model, 
Version  1.1  [Paulk  93b].  Training  in  the  reference  model  and  in  performing  an  SCE  is 
necessary  before  joining  an  SCE  team  as  an  evaluator. 


Document  Structure 

Part  1,  the  Introduction,  provides  background  information  about  the  purpose,  goals, 
objectives,  applications,  and  products  supporting  the  SCE  Method.  Part  2,  the  Overview  of  the 
SCE  Method,  provides  a  high  level  description  of  the  SCE  activities,  a  conceptual  framework 
for  CMM-based  data  collection,  information  about  various  SCE  application  characteristics, 
and  information  about  the  evolution  of  the  SCE  Method.  Part  3  includes  a  more  detailed 
description  for  users  of  the  method.  Tailoring  the  baseline  method  for  government  source 
selection  is  documented  in  the  SCE  Version  3.0  Implementation  Guide  for  Supplier  Selection. 

Appendices  include  a  SCE  V2.0  to  V3.0  activity  mapping,  a  CAF  requirements  traceability 
matrix,  temporal  flow  and  information  flow  diagrams,  a  baseline  schedule,  a  version  update 
description,  attribute  definitions,  a  glossary,  and  a  bibliography. 


Other  SCE  Documents 

In  addition  to  this  document,  the  initial  release  of  the  SCE  v3.0  document  suite  includes  an 
implementation  guide  for  supplier  selection.  These  documents  are  provided  as  part  of  team 
training.  Implementation  guides  for  other  uses  of  the  method,  such  as  internal  evaluation,  or 
with  models  similar  to  the  software  CMM  may  be  added  to  the  suite  upon  SEI  approval.  Also 
approved  with  initial  document  release  is  the  training  course  necessary  for  SCE  teams. 


|  SCE  Product  Suite  I 

Method  and  Guidance 

Education,  Training,  and  Qualification 

Transition  and  Installation 

•  Method  description  t 

•  Implementation  guide  for 
supplier  selection  t 

•  Implementation  guide  for 
process  monitoring 

•  Implementation  guide  for  internal 
evaluation 

•  Team  member  reference  guide 

•  Quick  reference  manual 

•  Evaluator  training  t 

•  Introduction  to  SCE 

•  Overview  seminar 

•  Senior  evaluator  training 

•  Refresher  training 

•  Qualification  program 

•  Reference  model  training 

•  Transition  strategy 

•  Installation  guide 

•  Technology  transition  workshop 

•  Implementation  workshop 

•  Communication  package 

•  Automated  support  aids 

f  indicates  that  an  item  is  part  of  the  initial  release  of  the  SCE  v3.0  product  suite. 

Table  P-1:  Proposed  SCE  Product  Suite 
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Introduction 


Software  Capability  Evaluation  Version  3.0  Method 

Description 


Abstract:  This  report  describes  Version  3.0  of  the  Software  Capability 
Evaluation  (SCE)  Method.  SCE  is  a  method  for  evaluating  the  software 
process  of  an  organization  to  gain  insight  into  its  process  capability. 
This  version  of  the  SCE  Method  is  based  on  the  Capability  Maturity 
Model  (CMM)  defined  in  Capability  Maturity  Model  for  Software, 
Version  1. 1  [Paulk  93a].  It  is  compliant  with  the  CMM  Appraisal 
Framework  (CAF)  [Masters  95].  This  document  is  an  update  to  SCE 
Version  2.0  [CBA  Project  94]. 


Part  1  Introduction 

This  part  of  the  document  contains  the  following  sections: 


Section  Page 

1.1  Background  and  Context  1 

1 .2  SCE  Goals  and  Objectives  5 

1 .3  Benefits  of  SCE  7 

1 .4  SCE  and  Other  SEI  Appraisal  Methods  8 

1 .5  Overview  of  the  SCE  Data  Collection  Model  11 


1.1  Background  and  Context 

The  principles  of  process  and  quality  management,  originally  proposed  by  Deming  and  Juran 
for  manufacturing  enterprises,  state  that  the  quality  of  a  system  is  largely  governed  by  the 
quality  of  the  process  used  to  develop  and  maintain  it  [Crosby  79,  Deming  86,  Humphrey  90, 
Juran  88],  The  validity  of  these  principles  has  been  demonstrated  by  many  successful 
manufacturers.  While  software  engineering  differs  significantly  from  manufacturing 
enterprises  in  some  respects,  the  fundamental  principles  of  product  quality  through  process 
improvement  can  be  effectively  transferred  across  these  domains.  By  application  of  these 
principles  to  the  software  domain,  the  SEI  has  developed  tools  such  as  the  "■►Software 
Capability  Evaluation  (SCE)  Method. 

SCE  is  a  method  for  evaluating  the  software  process1  of  an  organization  to  gain  insight  into 
its  "■••process  capability.  Process  capability  refers  to  the  range  of  expected  results  that  can 
be  achieved  by  following  a  process.  The  software  process  capability  of  an  organization 
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provides  one  means  of  predicting  the  most  likely  outcomes  of  an  organization’s  next  software 
project — for  example,  whether  the  software  will  be  produced  on  time  and  within  budget 
[Paulk93a].  SCE  provides  a  snapshot  of  an  organization’s  past  process  implementation, 
current  process  activities,  and  future  process  potential.  An  SCE  probes  the  organization’s 
process  implementation  to  establish  a  set  of  findings  used  to  support  the  business  needs  of 
the  '^sponsor. 

The  processes  evaluated  by  SCE  include  decision-making  processes  (such  as  project 
planning),  communication  processes  (such  as  intergroup  communication),  and  technical 
support  processes  (such  as  peer  reviews  and  product  engineering) — but  not  technical 
production  processes  (i.e.,  processes  required  by  a  particular  methodology,  such  as  object 
oriented  design),  as  shown  in  Figure  1-1. 
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1-  Software  process  is  defined  to  mean  the  system  of  all  tasks  and  the  supporting  tools,  standards,  methods,  and 
practices  involved  in  the  production  and  evolution  of  a  software  product  throughout  the  software  life  cycle. 
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SCE  (and  appraisals  generally)  are  only  one  aspect  of  a  broader  software  process 
improvement  effort.  Process  improvement  may  be  described  generically  by  models  such  as 
the  SEI-developed  IDEALSM2  approach  to  integrated  software  process  improvement.  The 
“••■IDEAL  model  is  a  systems  approach  or  life  cycle  framework  for  implementing  process 
improvement  activities.  IDEAL  stands  for  the  five  phases  of  the  approach:  Initiating, 
Diagnosing,  Establishing,  Acting,  and  Leveraging.  The  model  is  shown  in  Figure  1-2  on  page 


3. 


Learning 


Stimulus  (or  Change 


Establishing 


Figure  1-2:  IDEALSM  Approach  to  Software  Process  Improvement 


Depending  on  how  SCE  is  used,  it  might  be  found  in  the  initiating  and/or  diagnosing  phases 
of  IDEAL.  For  example,  in  Figure  1-2,  note  that  in  the  Initiating  phase,  some  outside  stimulus 
is  needed  for  process  improvement  to  occur.  An  external  SCE  conducted  “by”  one 
organization  “on”  another  organization  for  the  purpose  of  selecting  a  supplier  could  provide 
this  stimulus  and  start  an  organization  on  the  process  improvement  path.  The  actual  appraisal, 
using  the  SCE  method  to  characterize  the  current  practices  of  the  organization,  would  occur 
in  the  Diagnosing  phase.  Results  of  appraisals  are  used  to  develop  recommendations  for 
furthering  process  improvement  activities. 


2  IDEALsm  is  a  service  mark  of  Carnegie  Mellon  University 
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The  IDEAL  approach  assumes  that  the  appraisal  will  be  followed  by  the  development  of 
recommendations  and  a  thorough  documentation  of  the  appraisal  results. 


4 


©  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


CMU/SEI-96-TR-002 


April  1996 


Introduction 


1.2  SCE  Goals  and  Objectives 

The  overarching  goal  of  all  CMM-related  activities  is  disciplined  process  improvement.  This 
goal  is  stated  explicitly  for  CMM-Based  Appraisals  for  Internal  Process  Improvement  (CBA 
IPI),  but  it  is  also  implicitly  part  of  all  SCEs.  In  addition  an  SCE  meets  two  specific  business 
goals  for  its  sponsor. 

The  SCE  method  is  designed  to  support  organizations  in  appraising  business  processes.  SCE 
has  two  primary  goals: 

1 .  to  provide  results  that  support  senior  management  decision  making 

2.  to  obtain  accurate  results  relative  to  a  reference  model — the  status  of  the  organization’s 
existing  process  strengths,  weaknesses,  and  improvement  activities 

Each  of  these  goals  has  several  associated  objectives,  which  are  discussed  in  detail  below. 

Goal  1:  Provide  results  that  support  senior  management  decision  making.  To  provide 
business  value,  the  prime  objective  of  the  SCE  is  always  to  provide  results  that  support  senior 
management  decision  making  processes.  Some  key  objectives  are  the  following: 

•  The  sponsor  has  sufficient,  specific  data  upon  which  to  base  trade-offs  between  cost, 
schedule,  and  technical  factors  for  subsequent  actions. 

•  The  sponsor  has  sufficient,  specific  data  to  determine,  assess,  and  mitigate  risks  relative 
to  process  capability. 

•  The  sponsor  has  a  baseline  with  which  to  link  specific  business  goals  to  detailed  work  level 
processes. 

•  The  sponsor  has  sufficient  process  information  to  incorporate  into  larger  business 
acquisition  and  execution  decisions. 

Goal  2:  Obtain  accurate  results  relative  to  a  reference  model — the  status  of  the  organization’s 
existing  process  strengths,  weaknesses,  and  improvement  activities.  The  objectives  of  this 
goal  include  the  following: 

•  The  evaluation  team  provides  the  sponsor  with  the  data  needed  to  “baseline”  an 
understanding  of  the  organization’s  process  capability  and  to  track  improvements  over 
time. 

•  The  evaluation  team  identifies  process  strengths  on  which  the  organization  can  build  to 
improve  their  capability. 

•  The  evaluation  team  identifies  process  weaknesses  which  can  impact  the  organization’s 
performance. 

•  The  evaluation  team  identifies  the  major  non-reference  model  issues  that  have  an  impact 
on  successful  use  of  processes. 
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•  The  findings  are  sufficiently  complete  relative  to  the  reference  model  to  allow  the  sponsor 
to  prioritize  efforts  and  decisions. 

•  The  findings  are  sufficiently  granular  that  specific  process  action  plans  can  be  developed 
and  efforts  tracked. 

Organizations  also  need  an  accurate  picture  of  the  strengths  and  weaknesses  of  their  current 
processes  in  order  to  develop  process  action  plans.  Relating  these  strengths  and  weakness 
to  a  commonly  accepted  evolutionary  model  for  process  improvement  (e.g.,  CMM)  helps 
organizations  to  prioritize  their  plans  and  focus  on  those  improvements  that  are  most 
beneficial  given  their  current  level  of  maturity  and  their  business  goals.  This  demonstrates  the 
strong  interrelationship  of  the  goals. 

The  table  below  shows  the  relationship  between  SCE  Method  goals  and  its  primary 
application  areas.  A  “high-medium-low”  scale  has  been  used  to  show  how  the  goals  are 
emphasized  in  a  given  application.  Clearly,  this  information  is  conceptual.  All  SCE  uses  will 
be  tailored  to  emphasize  these  goals  differently  to  meet  sponsor  needs. 


Method  Goal  j 

1  Application  Type 

Obtain  Accurate  Results 

Support  Management  Decision  Making 

Supplier  selection 

medium 

high 

Process  monitoring 

medium 

high 

Internal  evaluation 

medium 

high 

Table  1-1:  SCE  Goal/  Application  Mapping 
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1 .3  Benefits  of  SCE 

An  SCE  helps  its  sponsor  make  better  business  decisions.  It  is  a  decision-making  tool  to 
identify  risks  inherent  in  a  software  development  organization’s  processes  that  may  inhibit 
achievement  of  the  sponsor’s  objectives.  It  also  measures  the  organization’s  commitment  to 
process  improvement. 

Process  improvements  typically  lead  to 

•  shorter  time  to  market 

•  increased  quality  (fewer  errors,  less  reworking) 

•  improved  customer  satisfaction 

These  factors  are  important  to  executives  and  managers  from  both  buyer  and  supplier 
perspectives.  Process  improvement  programs  both  in  and  outside  of  the  software  industry 
have  shown  returns  on  investment  between  4.5  to  1  and  7.5  to  1  [Herbsleb  94],  The  high  return 
on  investment  shows  the  potential  value  of  process  improvement  efforts,  and  provides 
motivation  for  a  buyer  to  focus  a  supplier’s  attention  on  improving  processes  or  for  a  supplier 
to  initiate  and  institutionalize  continuous  process  improvement  programs. 

A  CMM-based  SCE  will  be  appropriate  for  a  sponsor  who 

•  acquires  software  and  experiences  problems  with  the  cost,  quality,  or  schedule  (time  to 
market)  in  those  products 

•  needs  to  determine  risks  relative  to  a  product  development 

•  needs  to  evaluate  a  supplier’s  progress  in  an  internal  process  improvement  program 

•  needs  to  prioritize  efforts  for  efficient  allocation  of  resources 

•  wants  specific,  independent  feedback  about  processes  to  aid  a  higher-level  business 
decision-making  process 

While  an  SCE  must  first  and  foremost  meet  the  sponsor’s  needs  for  evaluation,  a  good 
appraisal  also  facilitates  subsequent  actions  within  the  evaluated  organization.  It  will  have 
maximum  impact  on  an  organization  if  the  organization  understands  customer  needs,  process 
improvement  principles,  has  a  strategy  for  improving  and  meeting  these  needs,  and  has 
established  specific  goals  and  objectives. 
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1 .4  SCE  and  Other  SEI  Appraisal  Methods 

1.4.1  Background 

In  accordance  with  the  Software  Engineering  Institute’s  mission  to  provide  leadership  in 
advancing  the  state  of  the  practice  of  software  engineering  by  improving  the  quality  of  systems 
that  depend  on  software,  there  has  been  strong  emphasis  on  software  development  tasks 
being  treated  as  processes  that  can  be  controlled,  measured,  and  improved.  In  an  early 
software  process  publication,  a  software  maturity  framework  was  developed  to  help 
organizations  characterize  the  current  state  of  their  software  practice,  set  goals  for  process 
improvement,  and  set  priorities  [Humphrey  87], 

The  SEI  assisted  a  number  of  organizations  in  performing  assessments  based  largely  on  the 
questionnaire  [Humphrey  87].  The  early  questionnaire  provided  a  scoring  mechanism  for 
determining  an  organization’s  maturity  level.  In  1988-91,  the  SEI  provided  training  to 
organizations  who  wished  to  perform  self-assessments  of  their  software  processes.  In  1990 
the  SEI  commercialized  the  software  process  assessment  (SPA)  to  more  broadly  disseminate 
the  technology.  Industry  and  government  licensees  were  selected  to  market  assessment 
services.  Data  from  these  assessments  have  been  collected  by  the  SEI,  and  reports  are 
delivered  on  a  regular  basis  reporting  the  state  of  the  practice  as  evidenced  by  assessment 
results  [Humphrey  89,  Kitson  92,  Zubrow  94b,  Zubrow  94c,  Zubrow  95]. 

The  original  version  of  the  SCE  Method  is  described  in  A  Method  for  Assessing  the  Software 
Engineering  Capability  of  Contractors  (CMU/SEI-87-TR-23).  It  was  developed  to  support 
source  selection  in  major  government  software  acquisitions.  While  the  major  activities  have 
remained  the  same,  other  aspects  of  the  SCE  Method  evolved  significantly  as  a  result  of 
feedback  from  users  of  the  method,  observing  the  effect  of  SCEs  on  industry,  and  the 
evolution  of  the  CMM  and  the  CAF.  This  led  to  public  baselining  of  the  SCE  Method  in  the  SCE 
Version  1 .5  Method  Description,  the  updating  of  the  method  to  comply  with  CMM  Version  1 .1 
in  SCE  Version  2.0  Method  Description,  and  to  the  changes  contained  in  this  document. 

Aligning  the  SEI  Methods  with  each  other  has  been  a  principal  “common  appraisal  goal”  of  the 
software  community  for  a  number  of  years.  Inputs  from  two  user  workshops  (1992, 1994)  and 
SCE  advisory  board  meetings  were  key  events  suggesting  this  need. 

1.4.2  Current  SEI  Appraisal  Methods 

To  provide  a  framework  for  rating  against  the  CMM  and  to  provide  a  basis  for  comparing 
assessment  and  evaluation  results,  the  SEI  established  the  CAF.  The  CAF  identifies  the 
requirements  and  desired  characteristics  of  a  CMM-based  appraisal  method  in  order  to 
improve  consistency  and  reliability  of  methods  and  their  results  [Masters  95].  The  term 
appraisal  as  used  at  the  SEI  includes  multiple  methods,  such  as  assessments  and 
evaluations,  all  of  which  focus  on  an  organization’s  software  development  process.  Both  CBA 
IP!  and  SCE  3.0  were  created  to  be  CAF-compliant. 
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CMM-Based  Appraisal  for  internal  Process  improvement  (CBA  IPI) 

The  CBA  IPI  Method  was  created  in  response  to  user  needs  for  a  CMM-based  assessment 
method.  The  SPA  Method,  which  has  become  so  familiar  in  the  software  community,  pre¬ 
dated  the  CMM.  Although  many  organizations  modified  the  SPA  to  reflect  the  CMM,  there  was 
a  wide  range  of  approaches  and  results. 

The  CBA  IPI  Method  was  developed  and  field  tested  in  1994.  After  factoring  in  lessons  learned 
from  the  community  feedback,  the  SEI  released  CBA  IPI  VI. 0  in  May  1995.The  CBA  IPI 
method  explicitly  uses  CMM  VI. 1  as  a  reference  model.  The  data  collection  is  based  on  key 
process  areas  of  the  CMM  as  well  as  non-CMM  issues.  CBA  IPI  is  intended  to  establish 
consistency  among  assessment  methods  so  that  results  from  one  assessment  can  be 
compared  to  those  of  another.  The  CBA  IPI  Method  complies  with  the  CAF,  so  results  from  a 
CBA  IPI  should  be  consistent  with  other  CAF-compliant  methods. 

Software  Capability  Evaluation  (SCE) 

SCEs  are  used  as  a  discriminator  to  select  suppliers,  for  contract  monitoring,  and  for 
evaluation  of  internal  processes.  SCE  V2.0  was  updated  to  reflect  CMM  VI. 1.  SCE  V3.0 
further  improves  the  method  to  comply  with  the  CAF  and  enhance  '^fidelity  and  "■•■reliability 
of  CBA  Methods.  Results  from  a  CAF-compliant  SCE  should  be  consistent  with  a  CBA  IPI  if 
the  areas  of  investigation  are  the  same  in  relatively  the  same  time  frame.  SCE  is  used  to  gain 
insight  into  the  software  process  capability  of  a  supplier  organization  and  is  intended  to  help 
decision  makers  make  better  decisions  when  choosing  from  among  suppliers,  improve 
subcontractor  performance,  and  provide  direction  to  a  purchasing  organization. 
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Differences  Between  Assessments  and  Evaluations 

The  primary  differences  between  assessments  and  evaluations  are  outlined  in  Table  1-2 
below. 


I  Assessments 

|  Evaluations  1 

Development  organization  uses  to  improve  processes 

Acquirers  use  in  selecting  and  monitoring  suppliers; 
developers  use  to  measure  improvement  progress 

Results  are  used  within  the  assessed  organization 

Results  are  known  to  sponsor 

Assesses  process  practice 

Substantiates  current  practice 

Acts  as  improvement  catalyst 

Evaluates  commitment  to  improve 

Provides  input  to  action  plan 

Analyzes  performance  potential  and  improvement 
input 

Collaborative — members  of  organization  must  be  on 
team 

Members  of  organization  may  or  may  not  be  on  team 

Applies  to  organization,  not  individual  projects  or 
contracts 

Organizational  data  applies  to  specific  sponsor  needs 

Input  for  improvement  action  plan  to  unfreeze 
organization 

Input  for  award  decision,  performance  monitoring,  risk 
management,  and  internal  improvement 
measurement 

Table  1-2:  Differences  Between  Assessments  and  Evaluations 


Interim  Profile 

The  Interim  Profile  (IP)  is  a  method  to  rapidly  measure  the  status  of  an  organization’s  software 
engineering  process  improvements  between  organizational  software  process  assessments 
[Hayes  95,  Whitney  94],  It  is  based  on  the  CMM  VI. 1  and  is  designed  to  be  used  only  by 
organizations  that  have  completed  an  SEI-style  assessment,  have  support  for  software 
engineering  process  improvement  in  place,  and  intend  to  use  the  results  to  monitor  status  and 
adjust  their  process  improvement  plan.  A  primary  motivation  for  designing  the  IP  method  was 
to  provide  quantifiable  data  in  an  expedient  fashion  to  allow  periodic  “temperature-checks”  of 
process  improvement  activities.  The  IP  method  is  based  on  the  SEI  maturity  questionnaire 
[Zubrow  94a]  with  minimal  use  of  other  data  sources.  The  IP  method  is  not  intended  to  comply 
with  the  CAF  since  it  focuses  on  questionnaire  responses  and  does  not  require  other  data 
collection  mechanisms  as  required  by  the  CAF.  it  is  a  “quick  look”  at  the  organization,  not  a 
full  assessment,  based  on  the  participants’  responses  to  the  questionnaire. 
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1.5  Overview  of  the  SCE  Data  Collection  Model 

As  mentioned  previously,  SCE  is  a  method  for  evaluating  the  software  process  of  an 
organization  to  gain  insight  into  its  process  capability.  To  evaluate  process  capability, 
sponsors  choose  teams  of  trained  and  experienced  people  from  their  organizations,  or 
contractors  trained  in  the  SCE  methodology.  The  team  uses  the  SCE  Method  to  sample  and 
analyze  information  about  the  organization’s  process  implementation.  SCE  is  a  model-based 
method;  this  means  that  the  information  sampled  and  analyzed  must  be  within  the  framework 
of  a  particular  model.  For  SCE,  this  is  the  CMM  vl  .1  [Paulk  93a].  In  addition  to  CMM  vl  .1 ,  the 
SCE  Method  uses  the  associated  key  practices  found  in  Key  Practices  of  the  Capability 
Maturity  Model,  Version  1.1  [Paulk  93b].  Key  practices  are  discussed  below. 

When  using  SCE  with  the  CMM  for  Software  VI  .1  as  the  reference  model,  an  SCE  team  may 
collect  data  on  process  capability  based  on  all  components  of  the  model,  including  maturity 
level,  Key  Process  Areas  (KPA),  goals,  common  features,  and  key  practices  (these  terms  are 
discussed  in  detail  below).  By  using  a  reference  model,  information  about  process  can  be 
systematically  organized  and  elaborated  in  a  way  that  facilitates  comparison  to  the  state  of 
practice  within  the  industry.  Non-reference  model  issues  that  may  have  a  major  impact  on  an 
organization’s  ability  to  implement  process  improvements  must  be  considered  along  with  the 
reference  model  related  issues. 

The  CMM  provides  a  robust  structure  for  collecting  information;  this  structure  is  diagrammed 
at  a  high  level  in  Figure  1-3.  Note  that  the  structure  is  not  strictly  hierarchical;  topics  include 
features  and  practices  associated  with  goals,  while  features  and  practices  may  be  associated 
with  more  than  one  topic  area  (this  perspective  can  work  equally  well  with  related  reference 
models  that  employ  the  same  architecture  as  the  CMM  for  Software,  such  as  the  Trusted 
Software  CMM). 
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Key  Process  Areas  (KPA) 

A  cluster  of  activities  that  achieve  a  set  of  goals. 


Common  Features 

A  set  of  attributes  that  indicate  the 
implementation  and  institutionalization 
status  of  a  KPA. 


Key  Practices 

Infrastructure  and  activities  that  contribute 
most  to  the  implementation  of  a  KPA. 


Topics 

A  focused  subject  matter  area 
probed  during  the  SCE 


investigation.^  is  a  subset  of  s 
process  activities  that  work  toward 
achieving  a  specific  KPA  goal. 
Topics  are  chosen  by  the 
intersection  of  process  area  goals 
and  its  associated  features. 


Topics  are  used  to  probe  the 
detailed  work  processes  actually 
implemented  by  an  organization. 


-  .  - . -  '• 


Figure  1-3:  The  SCE  CMM-Based  Data  Collection  Model 


Maturity  Levels.  As  noted  in  the  CMM  vl.1,  “continuous  process  improvement  is  based  on 
many  small,  evolutionary  steps  rather  than  revolutionary  innovations.  The  CMM  provides  a 
framework  for  organizing  these  steps  into  five  maturity  levels.”  A  -^maturity  level  is  a  “well- 
defined  evolutionary  plateau”  toward  achieving  a  mature  software  process.  Each  maturity 
level  provides  a  layer  in  the  foundation  for  continuous  process  improvement  [Paulk  93a].  The 
five  maturity  levels  of  the  CMM  are  Initial,  Repeatable,  Defined,  Managed,  and  Optimizing. 

Key  Process  Areas  (KPAs).  Except  for  the  Initial  level,  each  maturity  level  is  decomposed 
into  several  KPAs.  Key  process  areas  identify  the  issues  that  must  be  addressed  to  achieve 
a  maturity  level.  Examples  of  KPAs  include  requirements  management,  software  project 
planning,  and  software  configuration  management.  Each  KPA  identifies  a  cluster  of  related 
activities  that,  when  performed  collectively,  achieve  a  set  of  goals  considered  important  for 
enhancing  process  capability  [Paulk  93a]. 
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Each  KPA  contains  a  set  of  goals.  A  goal  describes  in  a  general  way  what  should  be  achieved 
by  implementing  a  KPA.  For  example,  the  goals  for  the  requirements  management  KPA  are 
the  following: 

•  System  requirements  allocated  to  software  are  controlled  to  establish  a  baseline  for 
software  engineering  and  management  use. 

•  Software  plans,  products,  and  activities  are  kept  consistent  with  the  system  requirements 
allocated  to  software. 

The  goals  can  be  used  to  determine  whether  an  organization  or  project  has  effectively 
implemented  the  KPA.  The  goals  signify  the  scope,  boundaries,  and  intent  of  each  KPA.  When 
evaluating  a  specific  implementation  of  a  KPA,  the  goals  can  be  used  to  determine  if  the 
implementation  satisfies  the  intent  of  the  KPA. 

The  path  to  achieving  the  goals  of  a  KPA  may  differ  across  projects  based  on  differences  in 
application  domains  or  environments.  Nevertheless,  all  of  the  goals  of  a  KPA  must  be 
achieved  for  the  organization  to  satisfy  that  KPA.  When  the  goals  of  a  KPA  are  accomplished 
on  a  continuing  basis  across  projects,  the  organization  can  be  said  to  have  institutionalized 
the  process  capability  characterized  by  the  KPA. 

Common  Features.  KPAs  are  organized  by  a  set  of  common  features.  A  common  feature  is 
“an  attribute  that  indicates  whether  the  implementation  and  institutionalization  of  a  key 
practice  is  effective,  repeatable,  and  lasting”  [Paulk  93b].  The  common  features  represent  the 
necessary  attributes  of  any  process. 

The  CMM  common  features  are  used  in  the  SCE  Method.  The  SCE  features  are  directly  from 
the  definitions  of  the  common  features  in  the  CMM.  A  feature  is  one  of  a  set  of  attributes  that 
provides  a  view  of  “whether  the  implementation  and  institutionalization  of  a  key  practice  are 
effective,  repeatable,  and  lasting”  [Paulk  93b].  The  features  are  more  appropriate  for  defining 
a  topic  of  investigation  than  the  common  feature  as  a  whole. 

"*■  Key  practices  are  the  infrastructure  and  activities  that  contribute  most  to  the  effective 
implementation  and  institutionalization  of  a  key  process  area  [Paulk  93b],  The  key  practices 
are  described  in  Key  Practices  of  the  Capability  Maturity  Model,  Version  1. 1  [Paulk  93b].  The 
key  practices  serve  as  examples  of  “what”  is  to  be  done;  they  should  not  be  interpreted  as 
mandating  “how”  the  goals  should  be  achieved.  Alternative  practices  may  accomplish  the 
goals  of  the  KPA.  The  key  practices  should  be  interpreted  to  judge  whether  the  goals  of  the 
KPA  are  achieved. 

The  key  practices  serve  as  examples  of  things  that  an  SCE  team  might  see.  They  are  helpful 
in  the  data  collection  and  consolidation  processes.  There  are  more  key  practices  in  the  CMM 
than  a  typical  team  can  investigate  thoroughly  in  a  typical  site  visit  (5  people  in  3  days).  That 
is  where  sampling  techniques  such  as  topic  selection  are  extremely  helpful. 
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SCE  is  a  model-based  method  that  provides  a  structure  for  collecting  information  at  varying 
levels  of  detail.  The  choice  of  information  to  be  sampled  in  an  SCE  is  determined  by  the 
•CMM  scope  of  the  investigation.  Often  the  high  level  scope  is  described  as  the  "••target 
process  capability — that  is,  the  process  capability,  in  terms  of  CMM  KPAs,  that  is  most 
appropriate  for  the  meeting  the  sponsor’s  business  objectives.  The  target  process  capability 
consists  of  a  set  of  KPAs  that  will  be  evaluated,  and  establishes  the  boundaries  of  the 
evaluation. 

Although  the  boundaries  of  the  SCE  are  determined  by  the  KPAs  defined  in  the  target  process 
capability,  the  evaluation  is  done  at  a  more  detailed  level.  A  process  area  is  a  set  of  activities 
in  an  implemented  process  that,  acting  together,  helps  an  organization  to  achieve  goals. 
There  are  two  or  more  goals  for  each  process  area;  each  goal  describes  something  that 
should  be  achieved  by  implementing  the  process  area.  However,  in  order  to  conduct  an  SCE 
investigation — that  is,  in  order  to  determine  what  documents  to  review,  whom  to  interview,  and 
what  kinds  of  questions  to  ask — teams  need  a  further  level  of  detail. 

The  level  of  detail  at  which  an  SCE  is  conducted  is  the  «•  topic.  A  topic  defines  a  subject 
matter  area  that  will  be  probed  by  the  team  during  an  SCE.  A  topic  can  be  transformed  into 
open-ended  questions  that  can  be  readily  answered  by  a  person  in  an  interview  or  that  can  be 
validated  by  the  team  in  a  review  of  documentation.  For  example,  within  the  CMM  Software 
Project  Planning  KPA,  there  is  a  goal  related  to  developing  estimates.  The  team  might  want 
to  investigate  what  procedures  are  in  place  to  help  ensure  practitioners  derive  estimates  in  a 
similar  manner  (the  activities  performed  common  feature).  Thus,  they  could  ask  the  question, 
“What  are  the  procedures  used  to  develop  software  size  estimates?”  The  topic  investigated  is 
estimation  procedures. 

Topics  are  not  a  part  of  the  CMM  structure,  but  rather  are  an  SCE  method-specific  technique 
for  maintaining  consistency  with  the  CMM  while  allowing  the  team  to  best  meet  appraisal 
requirements.  Topics  are  generated  by  looking  at  the  intersection  of  a  KPA  goal  with  a 
common  feature.  This  intersection  creates  a  subject  matter  area  upon  which  the  team  can 
develop  appropriate  data  collection  strategies.  Topics  help  to  solve  the  team’s  dilemma  of 
“looking  for”  implementation  details  of  actual  process  use,  yet  needing  to  judge  that  detailed 
data  against  very  high  level  goals.  Topics  help  bridge  this  “judgment  gap”  inherent  in  the 
model  structure. 

Example.  The  following  example  uses  the  Software  Project  Planning  KPA  to  illustrate  the 
relationships  among  the  concepts  described  above. 

When  it  has  implemented  the  Software  Project  Planning  KPA,  an  organization  will  be  able  to 
establish  reasonable  plans  for  performing  software  engineering  and  for  managing  the 
software  project. 
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Within  this  KPA,  one  of  the  things  the  organization  hopes  to  accomplish  is  to  make  sure  that 
affected  groups  and  individuals  agree  to  their  commitments  related  to  the  software  project  (a 
goal).  The  SCE  Method  uses  the  shorthand  term  make  commitments  to  describe  the  activities 
performed  to  achieve  this  goal. 

An  example  of  one  of  these  activities  is  ensuring  that  software  project  commitments  made  to 
individuals  and  groups  external  to  the  organization  are  reviewed  with  senior  management 
according  to  a  documented  procedure  (a  key  practice).  This  key  practice  is  categorized  as  one 
of  the  Activities  Performed  within  the  Software  Project  Planning  KPA. 

A  topic  for  investigation  by  the  team  might  then  be,  “What  policies  and  procedures 
(commitment  to  perform  common  feature)  are  implemented  to  help  institutionalize  this 
activity?” 
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This  part  of  the  document  provides  an  overview  of  the  activities  executed  in  the  SCE  Method, 
taking  a  high  level  perspective  of  the  three  phases:  Plan  and  Prepare  For  Evaluation,  Conduct 
Evaluation,  and  Report  Evaluation  Results.  A  figure  highlighting  key  points  in  the  process  is 
provided,  followed  by  a  description  of  activities  in  each  phase.  A  brief  description  of  the 
baseline  SCE  Method  and  its  application  is  provided  to  close  out  this  section. 

To  perform  an  appraisal  using  SCE,  sponsors  choose  teams  of  trained  and  experienced 
people  from  their  organizations,  or  contractors  trained  in  the  SCE  Methodology.  The  team 
uses  the  SCE  Method  to  sample  and  analyze  information  about  the  organization’s  process 
implementation. 

There  are  four  major  ways  information  is  collected:  interviewing,  "^document  review,  «► 
presentations,  and  «►  instruments.  These  are  discussed  in  detail  below. 

The  analysis  and  summary  of  the  information  collected  on  an  SCE  becomes  the  findings 
of  the  team.  Findings  document  the  process  strengths,  weaknesses,  and  improvement 
activities  in  the  process  areas  evaluated  by  the  team.  An  «*■  improvement  activity  is  a 
process  improvement  that  is  not  yet  institutionalized — for  example,  a  pilot  of  a  new  process 
put  in  place  to  address  a  weakness  identified  by  the  organization.  Findings  are  the  "*•  results 
of  the  evaluation.  Findings  are  used  by  the  evaluation  sponsor  to  determine  risk  from  the 
implemented  processes  relative  to  specific  business  goals  (such  as  a  planned  development, 
a  new  product  line,  business  competitiveness,  progress  against  an  improvement  plan,  etc.). 
How  the  findings  (results)  are  used  by  the  sponsor  represents  the  ■►outcome  of  the 
evaluation. 
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The  SCE  Method  consists  of  three  major  activity  phases:  Plan  and  Prepare  for  Evaluation, 
Conduct  Evaluation,  and  Report  Evaluation  Results  (Figure  2-1). 


SCE  V3.0  Phase 


Plan  and  Prepare 
for  Evaluation 


Conduct 

Evaluation 


Report  Evaluation 
Results 


Activities  and  Outcomes 


The  sponsoring  organization 

•  Determines  the  desired  product  attributes. 

•  Determines  the  process  capability  that  is  most  appropriate  for  the 
business  goals  (the  Target  Process  Capability). 

•  Selects  and  trains  the  SCE  team. 

Outcome :  Evaluation  goals  and  requirements  are  defined. 

The  SCE  team 

•  Identifies  areas  where  the  development  organization(s)  lack 
experience  (indicating  potential  risk). 

•  Defines  the  scope  of  the  evaluation. 

Outcome:  Evaluation  scope  defined  and  high-level  preparations  for 
evaluating  the  development  organization(s)  completed. 


The  SCE  team 

•  Selects  projects  for  evaluation. 

•  Prepares  specific  topics  for  evaluation. 

•  Analyzes  instrument  data. 

Outcome:  Detailed  preparations  for  evaluating  a  particular 
development  site  completed. 


The  SCE  team 

•  Investigates  each  planned  topic  area  on-site. 

•  Conducts  data  collection  activities  through  interviewing,  document 
review,  and  presentations. 

•  Consolidates  the  information  collected  and  validates  observations. 

•  Determines  strengths,  weaknesses,  and  improvement  activities. 

Outcome:  Process  data  consolidated  and  findings  determined. 

The  SCE  team 

•  Presents  and  delivers  the  findings  to  the  sponsor  and  organization. 

•  Produces  a  final  report  for  the  sponsor. 

•  Makes  recommendations  for  use  of  the  findings. 

Outcome:  Evaluation  results  documented  and  outcomes  determined. 


Figure  2-1 :  Phases  and  Activities  of  an  SCE 
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2.1  Plan  and  Prepare  For  Evaluation  Phase 

In  this  phase,  the  sponsoring  organization  decides  to  use  the  SCE  Method,  and  prepares  for 
the  evaluation.  This  phase  is  jointly  performed  by  the  sponsor  and  the  SCE  team. 

During  this  phase,  the  sponsor  determines  the  role  of  SCE,  determines  the  attributes  of 
the  desired  product  and  the  project  required  to  produce  it,  determines  the  process  capability 
that  is  most  appropriate  to  meet  the  business  goals,  and  selects  the  SCE  team.  These 
activities  link  the  SCE  process  and  the  system  level  process  that  uses  the  SCE  findings  (such 
as  a  supplier  selection  process). 

The  focus  of  the  SCE  is  on  risks  associated  with  process  capability.  The  sponsoring 
organization  considers  how  SCE  can  be  used  in  conjunction  with  other  technical  and 
managerial  activities  to  identify  and  mitigate  risks  associated  with  the  target  processes  --  those 
processes  that  must  be  executed  well  to  deliver  a  quality  product  of  interest  on  time,  within 
budget.  Risks  not  associated  with  process  capability  may  also  be  captured  as  a  by-product  of 
conducing  an  SCE  (such  as  non-CMM  findings)  and  delivered  to  the  sponsor. 

With  this  in  mind,  the  sponsoring  organization  should  define  how  SCE  results  will  be  used  and 
should  determine  the  resources  required  to  perform  the  SCEs.  At  some  point  early  in  this 
phase,  the  sponsoring  organization  makes  a  formal  decision  to  use  the  SCE  Method. 

During  this  phase,  planning  for  the  SCE  should  consider 

•  funding  for  personnel,  training,  and  travel 

•  coordinating  SCE  "*•  site  visits  and  requests  for  information  with  the  development 
organization(s) 

•  scheduling  time  for  the  SCE  activities  within  the  context  of  the  use  of  the  method 
(e.g.,  supplier  selection,  process  monitoring,  or  internal  evaluation) 

As  a  result  of  this  planning,  the  sponsoring  organization  commits  resources  to  conduct  the 
SCE.  Determining  the  role  of  SCE  and  planning  for  the  use  of  the  method  may  not  be  done  by 
the  SCE  team,  but  these  activities  are  critical  to  the  successful  use  of  the  SCE  Method. 

Once  the  decision  to  use  SCE  is  made,  the  sponsoring  organization  determines  the  process 
capabilities  needed  to  meet  business  objectives.  This  is  accomplished  by  analyzing  the 
attributes  of  the  desired  product  and  then  determining  the  process  capability  that  is  most 
appropriate  for  building  the  desired  product.  The  processes  examined  by  an  SCE  always  fall 
within  the  bounds  of  the  CMM  reference  model,  although  non-model  related  findings  may  also 
be  noted  and  reported.  The  Target  Process  Capability  establishes  the  boundaries  of  the  SCE 
investigation — a  process  area  is  evaluated  if  and  only  if  it  is  part  of  the  Target  Process 
Capability. 
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The  sponsoring  organization  also  selects  the  SCE  team.1  A  team  consists  of  experienced 
people  who  have  completed  SCE  Evaluator  training.  They  may  come  from  the  sponsoring 
organization  or  a  contractor  specializing  in  evaluation  services  selected  by  the  sponsor. 
Senior  management  should  select  the  SCE  team  and  assign  the  personnel  resources,  and 
should  assess  the  potential  impact  on  schedule.  Staff  with  the  appropriate  engineering 
experience  should  establish  the  Target  Process  Capability  for  the  SCE. 

During  the  Plan  and  Prepare  for  Evaluation  phase,  the  team  defines  the  scope  of  the 
investigation  for  all  development  organizations  to  be  evaluated.  This  set  of  activities  is 
particularly  important  for  successful  supplier  selection  applications  of  the  SCE  Method.  The 
CMM  scope  of  the  SCE  consists  of  those  CMM  components  chosen  to  be  investigated  within 
the  Target  Process  Capability. 

The  SCE  team  identifies  those  processes  that  contribute  most  to  the  potential  development 
risk  throughout  the  development  organization  community.  It  then  examines  information  from 
each  development  organization  about  their  view  of  the  product  to  be  built  and  information 
about  the  projects  they  are  submitting  as  candidates  for  evaluation.  The  attributes  of  the 
product  to  be  built  are  compared  to  the  attributes  of  products  developed  by  the  projects  that 
have  been  submitted  as  candidates  for  evaluation.  These  comparisons  identify  areas  in  which 
the  development  organization  may  lack  experience,  indicating  potential  risk. 

Potential  risk  areas  identified  by  examining  the  experience  of  individual  development 
organizations  are  then  consolidated  for  all  of  the  organizations  to  be  evaluated.  Based  on  the 
experience  shortfalls  in  the  development  organization  community  the  SCE  team  selects 
components  for  evaluation  from  within  each  process  area  in  the  Target  Process  Capability. 
These  areas  will  be  investigated  at  all  development  organization  sites.  Collectively,  the 
specific  process  areas  chosen  for  investigation  define  the  CMM  scope  of  the  SCE. 

The  areas  selected  define  the  scope  of  the  SCE.  The  attributes  defined  by  the  sponsor  in  the 
product  profile  was  previously  used  to  establish  the  boundaries  of  the  investigation  in  terms  of 
process  areas;  the  collective  experience  of  the  development  organization  community  is  used 
to  help  the  team  focus  their  investigation  within  those  boundaries  and  prioritize  their  time.  This 
tailoring  is  necessary  because  of  site  visit  time  limitations.  During  a  site  visit,  sampling 
techniques  are  used  to  investigate  the  areas  within  the  Target  Process  Capability.  The  scope 
determination  is  a  way  of  limiting  the  sample  space  in  order  to  meet  other  important  evaluation 
objectives,  such  as  time  on  site  and  cost  of  the  evaluation. 


1  The  sponsoring  organization  always  has  the  responsibility  for  team  selection.  The  sponsor  may  choose  to 
delegate  team  selection  to  a  third  party.  An  example  is  when  the  objective  is  to  obtain  an  independent 
evaluation  for  the  purpose  of  preparing  for  an  upcoming  customer  led  SCE. 
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The  activities  described  here  apply  primarily  to  use  of  the  SCE  Method  in  supplier  selection, 
where  multiple  development  organizations  are  evaluated.  This  preparation  sets  a  level  playing 
field.  In  all  cases,  the  determination  of  CMM  scope  is  made  during  the  Plan  and  Prepare  For 
Evaluation  phase. 

In  process  monitoring,  the  same  activities  are  followed  for  the  initial  evaluation,  assuming  no 
contract  has  been  awarded  yet,  or  has  just  been  awarded.  This  initial  evaluation  creates  the 
initial  process  baseline.  Subsequent  evaluations  of  the  same  organization,  from  a  process 
monitoring  perspective,  would  be  tailored  to  reflect  the  special  needs  of  the  contract  and  the 
weaknesses  observed  during  the  initial  baseline  evaluation.  Since  there  is  only  one  contract, 
a  process  monitoring  SCE  does  not  execute  activities  related  to  an  entire  community  of 
development  organizations.  Similarly,  since  there  is  a  contract,  the  process  monitoring  SCE 
may  focus  entirely  on  the  project  at  hand,  rather  than  on  evaluating  mismatches  of  multiple 
projects. 

The  activities  described  here  that  are  related  to  an  entire  community  of  development 
organizations  are  typically  not  executed  in  an  internal  evaluation  SCE  application,  because 
only  one  organization  is  being  evaluated.  Mismatches  in  experience  would  reflect  just  one 
organization. 

The  SCE  team  then  prepares  for  a  specific  site  visit.  The  SCE  team  selects  projects  to 
evaluate,  people  to  interview,  and  topics  for  investigation.  These  activities  also  are  key 
sampling  measures  that  can  be  tailored  for  an  evaluation.  To  do  this,  the  team  selects  topic 
areas  from  within  the  CMM  scope.  Topics  address  observable  work  practices  and  are  used  to 
probe  the  process  implementation  that  corresponds  to  the  process  areas  identified  for 
investigation.  A  «*■  topic  defines  a  specific  subject  matter  area  that  will  be  probed  during  the 
investigation.  For  example,  a  topic  might  be,  “investigate  whether  the  organization  has 
standard  procedures  for  the  software  configuration  management  change  control  process.” 
Topics  are  developed  by  considering  ■*  features;  features  are  implementation  characteristics 
that  are  common  to  every  mature  process.  The  features  used  in  the  SCE  Method  are  directly 
from  the  '“♦common  features  of  CMM  vl  .1  [Paulk  93a]  and  are  described  in  the  glossary.  For 
example,  every  process  should  have  corresponding  training  and  should  also  have 
documented  plans  and  procedures;  “training”  and  “plans  and  procedures”  are  two  of  the 
features  that  can  be  used  to  develop  topics  for  investigation.  They  directly  relate  to  the 
commitment  to  perform  and  ability  to  perform  common  features  of  the  CMM  for  Software.  The 
intersection  of  a  matrix  mapping  the  goals  (and  activities)  selected  (implementation)  with  the 
common  features  (■*•  institutionalization)  generates  a  topic  area. 

After  selecting  evaluation  topics,  the  team  creates  a  strategy  for  conducting  interviews  and 
reviewing  documents.  The  team  then  works  closely  with  the  organization’s  site  technical 
coordinator  to  schedule  interviews,  request  documentation  for  review,  and  to  make  facility 
arrangements. 
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When  the  Plan  and  Prepare  For  Evaluation  phase  is  finished,  the  SCE  team  will  be  ready  to 
perform  the  site  visit.  The  team  will  have  determined  what  topics  will  be  investigated  (and  to 
what  level),  whom  they  need  to  talk  to,  what  initial  questions  they  need  to  ask  during 
interviews,  and  which  documents  they  will  review  first.  The  development  organization  will  have 
prepared  the  facility  for  the  team,  will  have  the  requested  documentation  on  hand,  and  will 
have  ensured  that  the  interviewees  are  available.  Thorough  preparation  is  essential,  because 
the  amount  of  information  to  be  considered  during  the  site  visit  will  overwhelm  the  SCE  team 
members  if  they  are  not  sufficiently  prepared. 
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2.2  Conduct  Evaluation  Phase 

The  purpose  of  the  Conduct  Evaluation  phase  is  to  investigate  the  topics  associated  with  each 
key  process  area  selected  in  enough  depth  to  determine  the  strengths,  weaknesses  and 
improvement  activities  of  those  areas.  Although  the  purpose  is  simple,  this  is  the  most 
complicated  activity  during  an  SCE. 

To  successfully  complete  the  investigation,  the  team  needs  to  have  a  good  working 
relationship  with  the  development  organization’s  site  visit  coordinator.  This  relationship  builds 
on  the  previous  contacts  with  the  site  visit  coordinator  made  during  the  preparation  activities. 
The  team  should  also  maintain  high  standards  of  professional  conduct  (e.g.,  maintain 
schedule,  remain  attentive,  practice  active  listening,  etc.).  This  helps  to  establish  their 
credibility  and  to  increase  the  level  of  cooperation  they  receive  from  development  organization 
personnel. 

After  setting  expectations  for  the  site  visit  with  an  entry  briefing,  the  team  starts  the  data 
collection  activities.  Site  data  collection  has  two  basic  components:  investigation  of  the  topics 
and  decision  making  about  the  information  collected.  These  components  are  applied 
iteratively  until  a  decision  has  been  made  about  each  topic  under  investigation;  this  is 
summarized  in  Figure  2-2. 
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The  SCE  team  uses  complementary  mechanisms  to  investigate  a  topic  during  the  on-site 
period:  document  review,  interviews,  and  presentations.  Instruments  (such  as  the  maturity 
questionnaire),  are  used  as  information  sources  during  the  plan  and  prepare  phase. 

Documents  can  be  used  to  define  and  standardize  processes,  indicate  commitment  to  use  the 
processes,  provide  an  audit  trail  of  processes  that  were  used,  and  collect  data  about  process 
performance.  Reviewing  documents  can  provide  objective  evidence  of  the  processes  used.  A 
fundamental  assumption  of  the  SCE  Method  is  that  an  executed  process  will  produce  artifacts, 
both  intermediate  products  and  end  products.  These  artifacts  can  be  investigated  to  determine 
the  extent  of  implementation  across  a  site. 

Interviews  give  insight  into  how  the  processes  are  implemented  in  practice  and  show  the 
extent  that  processes  are  internalized  and  understood  by  the  development  organization  staff. 
There  are  two  types  of  interviews  used  during  an  SCE:  exploratory  interviews  and 
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consolidation  interviews.  During  exploratory  interviews  the  questions  and  answers  reveal  the 
actual  processes  practiced  and  guide  the  team  to  the  supporting  documentation. 
Consolidation  interviews  focus  on  corroboration  and  clarification  of  evidence. 

Presentations  provide  both  additional  data  (when  the  organization  presents  to  the  team,  and 
when  participants  react  to  team  presentations)  and  validation  of  data  (when  participants  react 
to  preliminary  observations). 

By  using  multiple  data  gathering  mechanisms,  the  quality  of  the  data  is  improved.  For 
example,  questionnaires  are  used  to  gather  initial  information  in  a  consistent  and  economical 
manner.  The  information  is  used  to  guide  the  interviewing  process,  which  can  be  quickly  re¬ 
focused  on  areas  of  interest.  Document  review  is  used  to  supplement  the  interviewing 
process;  this  eliminates  most  of  the  recall  error  inherent  in  interviewing.  Later,  the  draft 
findings  presentation  is  used  to  feed  back  what  the  team  observed  to  the  organization, 
allowing  the  organization  a  chance  to  correct  errors.  The  quality  resulting  from  using  multiple 
data  collection  mechanisms  in  combination  is  much  better  than  the  team  could  achieve  using 
any  single  method. 

Interviews  are  a  flexible  way  of  eliciting  a  lot  of  information  quickly.  However,  interviewing  has 
potential  errors  due  to  human  recall  and  to  failures  in  the  communications  process.  Recall 
errors  are  minimized  by  asking  the  interviewees  to  provide  work  products  and  documentation 
relevant  to  their  job.  Providing  the  documentation  reduces  errors  by  focusing  the  interviewees 
on  items  that  can  be  substantiated.  To  mitigate  potential  errors  from  the  communication 
process,  the  team  interviews  as  a  group,  takes  extensive  notes,  and  compares  their 
observations  during  the  consolidation  process. 

The  team  members  record  the  data  collected  first  as  notes.  The  data  is  transformed  into 
observations  relative  to  the  reference  model  used.  These  observations  are  consolidated 
through  a  team  consensus  process  in  an  ongoing  team  «*■  caucus.  In  a  caucus,  the  individual 
team  members,  “mini-teams,"  and  often  the  entire  team  ask  the  question,  “Do  we  have  enough 
information  about  this  topic  to  make  a  judgment?”  Consensus  is  an  ongoing  process  at  various 
level  of  detail  within  the  data  consolidation  activities.  The  team  must  agree  that  there  are  at 
least  two  validated  observations  in  order  to  make  a  judgment  about  that  process  component. 
If  the  evidence  is  not  sufficient  or  conclusive,  a  new  round  of  interviewing  and/or  document 
review  is  planned  and  initiated. 

Validated  observations  become  candidates  for  findings  of  strengths,  weaknesses,  or 
improvement  activities  associated  with  one  or  more  of  the  topics  under  investigation.  The  team 
rates  the  satisfaction  of  specific  reference  model  components  that  were  decided  to  be  rated 
during  planning,  and  are  fully  covered  (corroborated,  validated)  during  the  site  visit.  Rating 
judgments  are  always  made  by  the  whole  team  in  a  consensus  process.  The  determination  of 
findings  and  determination  of  ratings  are  integrally  linked  but  are  separate  decision  processes 
in  the  evaluation. 
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When  the  Conduct  Evaluation  phase  is  finished,  the  SCE  team  members  are  ready  to  produce 
reports  for  delivery  to  the  sponsor  and  the  recipient  organization. 


26 


©  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


CMU/SEI-96-TR-002 


April  1996 


Overview  of  the  SCE  Process 


2.3  Report  Evaluation  Results  Phase 

The  Report  Evaluation  Results  phase  completes  the  SCE.  During  this  phase,  the  SCE  team 
documents  the  results  of  the  investigation  and  delivers  it  to  the  sponsor  and  the  appraised 
organization.  Subsequent  to  the  delivery  of  the  evaluation  reports  and  the  proper  disposition 
of  the  data,  the  team  (or  individuals  from  the  team)  supports  the  sponsor  in  follow  on  activities 
which  incorporate  use  of  the  findings  into  the  larger  system  for  which  the  sponsor  chartered 
the  SCE. 

Because  of  the  importance  of  the  SCE  findings  to  process  improvement,  efforts  should  be 
made  to  provide  feedback  in  a  complete  and  timely  manner.  Ideally,  the  SCE  team  presents 
the  findings  to  the  appraised  organization(s)  at  the  conclusion  of  each  site  visit.2  The  findings 
briefing  will  include  all  of  the  findings  determined  by  the  team,  any  ratings  that  were  generated, 
and  will  describe  what  will  happen  with  the  results. 

A  final  report  is  usually  completed  by  the  team  members  after  the  site  visit,  and  it  becomes  the 
actual  baseline  of  all  evaluation  activities  and  results  for  future  use  by  the  sponsor.  The  final 
report  may  include  recommendations,  if  desired  by  the  sponsor.  Often,  the  recommendations 
are  done  in  a  follow-up  activity  and  documented  separately.  Reports  are  also  generated  for 
the  SEI  process  database  and  for  the  method  developer  for  future  improvement  of  the 
evaluation  process. 

Delivery  of  the  findings  and  other  reports  is  the  distinct  transfer  of  evaluation  team  activities 
back  into  the  domain  of  the  sponsor  activities  (the  follow-on  activities).  When  the  Report 
Evaluation  Results  phase  is  complete,  a  formal  final  report  will  be  generated  for  the 
sponsoring  organization  to  use.  How  the  findings  report  is  used  depends  on  the  particular 
application  of  the  method. 

SCE  team  members,  and  certainly  the  team  leader,  can  expect  to  participate  in  follow-on 
activities.  The  knowledge  they  have  gained  during  the  SCE  is  very  valuable  to  the  sponsor  in 
deciding  what  actions  to  take  based  on  the  SCE  results.  In  supplier  selection,  this  support  to 
the  sponsor  is  usually  assisting  the  team  in  identifying  the  risks  related  to  the  specific  findings. 
In  a  process  monitoring  SCE,  this  support  might  include  assisting  the  sponsor  in  jointly 
producing  an  improvement  plan  with  the  supplier  organization,  or  in  focusing  an  award  fee 
plan.  In  an  internal  evaluation,  the  follow-on  support  might  be  in  forming  recommendations, 
participating  in  actions  teams,  consulting  with  the  sponsor,  or  other  forms  of  improvement 
planning  and  implementation. 


2  In  some  cases,  the  government  source  selection  authority  may  not  allow  the  findings  to  be  presented  to  the 
development  organization,  or  may  specify  that  findings  be  presented  after  contract  award  such  that  the 
immediate  findings  briefing  presented  by  the  team  is  little  more  than  an  exit  briefing. 
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2.4  Baseline  SCE  V3.0  Method 

The  SCE  Method  can  be  described  in  terms  of  attributes  that  are  tailored  or  modified  to  meet 
various  needs.  SCE  V3.0  is  designed  to  support  a  variety  of  needs,  and  is  expected  to  be 
tailored,  modified,  and/or  extended  to  help  achieve  business  goals.  Throughout  this 
document,  examples  from  various  SCE  applications  are  used  which  reflect  tailoring  of 
appraisal  attributes. 


Figure  2-3  provides  a  graphic  summary  of  the  major  attributes  which  can  be  tailored.  The 
major  items  include:  primary  diagnostic  purpose,  organizational  scope,  team  size/time  on  site, 
and  model  scope.  SCE  V3.0  is  a  specific  instance  resulting  from  tailoring  these  items.  As  can 
be  seen  from  the  figure,  the  baseline  SCE  V3.0  Method  is  intended  to  be  a  “mid-point” 
diagnostic.  This  is  consistent  with  the  SCE  V2.0  Method.  The  CBA  IPI  would  be  in  the  upper 
right  hand  corner  as  it  is  a  motivational  breadth-oriented  appraisal  that  generally  has  an 
organizational  scope. 

The  baseline  SCE  V 3.0  Method  is  a  specific  tailoring  of  these  concepts  to  achieve  method 
goals  described  in  Part  1  of  this  document.  The  principal  assumptions  built  into  the  “baseline” 
method  (when  used  for  supplier  selection)  are  listed  below. 
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Diagnostic  Purpose 

•  a  “mid-point”  diagnostic —  combines  both  “wellness”  (status  check)  and 
“motivation”  -oriented  characteristics 


Organizational  Scope 

•  one  appraised  entity —  single  location  (per  site  visit) 

•  investigation  is  application  domain  or  product  line  oriented 

•  three  current  projects  with  similar  characteristics 

•  proposed  project  is  evaluated 

CMM  Scope 

Either 

•  entire  specified  model  component  is  investigated  (complete  coverage)  (e.g.,  if  a 
particular  KPA  goal  is  investigated,  all  activities  performed  key  practices  mapped 
to  that  goal  are  investigated), 

or 

•  subset  of  reference  model  components  is  selected  (partial  scope)  (e.g.,  four  of 
seven  level  three  KPAs  in  the  CMM  for  Software,  VI  .1) 

•  investigation  is  “depth  oriented”  —  a  subset  of  model  items  is  covered  in  detail 

•  rating  baseline  option  is  “depth  oriented  —  partial  scope,  complete  coverage” 

•  scope  specifies  investigation  of  practices 

•  ratings  are  determined  (and  provided  to  the  sponsor) 

•  maturity  level  rating  is  not  done  (lack  of  complete  KPA  coverage) 

•  KPAs,  goals,  and  key  practices  may  be  rated  (if  selected  for  rating  during  planning, 
and  sufficient  coverage  attained) 

Team  Traits 

•  five  team  members 

•  all  are  external  to  the  evaluated  entity 

•  all  are  trained  in  the  reference  model  and  method 

•  all  meet  individual  and  aggregate  SCE  and  CAF  experience  requirements 

Time  on  Site,  Intervention  Techniques,  and  Artifacts  Used 

•  three  to  three  and  one  half  days  on  site  (actual  “disruption”  period  to  the  appraised 
entity) 
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•  both  manager  and  practitioner  interviews  are  conducted 

•  draft  findings  are  presented  to  participants  for  feedback  if  allowed  by  sponsor 

•  evaluation  results  —  findings  and  ratings  —  are  provided  on-site  if  allowed  by 
sponsor 

•  automatable  forms  and  instruments  used 

See  Section  4.2,  Develop  Evaluation  Plan,  for  further  discussion  on  method  tailoring 
decisions.  Alternate  applications  of  the  SCE  Method  are  defined  by  making  tailoring  decisions 
from  this  baseline  approach.  The  attributes  can  be  thought  of  as  sliding  scales.  The  team 
tailors  the  attributes  to  achieve  sponsor  goals. 


2.5  Application  Characteristics 

SCE  can  be  tailored  based  on  specific  business  and  evaluation  goals  such  that  the  results  are 
focused  on  the  information  most  pertinent  to  their  intended  use  by  the  sponsor.  There  are 
three  primary  application  areas  of  the  method: 

•  Supplier  selection.  SCE  was  originally  created  for  use  in  government  source 
selections  within  the  Department  of  Defense.  In  a  software  intensive  project,  SCE 
is  a  high  value  discriminator  used  to  select  from  among  suppliers.  In  source 
selection,  the  results  of  the  SCE  are  used  to  characterize  the  process-related  risk 
of  awarding  a  contract  to  an  offeror.3  SCE  is  typically  only  one  criterion  among 
many  used  to  select  contractors.  Applications  include  both  prime  contractor 
selection  and  subcontractor  selection. 

•  Process  monitoring.  SCE  has  also  been  used  in  monitoring  contractor  processes 
(for  example,  after  contract  award  by  serving  as  an  input  for  an  incentive/award  fee 
decision).  SCE  has  also  been  used  to  help  the  sponsoring  organization  tailor  its 
contract  monitoring  efforts  by  allowing  it  to  prioritize  efforts  based  on  the  observed 
strengths  and  weaknesses  of  the  development  organization’s  processes.  These 
uses  focus  on  a  long-term  teaming  relationship  between  the  sponsoring 
organization  and  the  development  organization. 

•  Internal  evaluation.  SCE  is  also  used  by  companies  to  provide  independent 
evaluations  of  their  internal  processes.  Applications  include  measuring  process 
improvement  progress,  conducting  process  audits  to  prepare  for  competitions  that 
may  include  external  SCEs,  focusing  on  domains  or  product  lines,  and  other 
project-specific  evaluations.  In  this  application  SCE  supplements  other  tools,  such 
as  CBA  IPI,  for  appraising  process  improvement  activities. 


3  Because  SCE  has  been  used  extensively  in  source  selection,  in  the  SCE  team  training  handouts  and  case 
study  materials  the  terms  offeror  and  contractor  are  often  used  to  denote  the  development  organization.  The 
development  organization  is  the  recipient  of  the  SCE.  Similarly,  in  the  training  materials  the  term  «»acquisi- 
tion  agency  is  often  used  to  denote  the  '^sponsoring  organization,  which  is  the  organization  conducting  the 
SCE.  This  document  uses  the  terms  development  organization  and  sponsoring  organization  almost  exclusive¬ 
ly  in  anticipation  of  wider  use  of  the  method  in  other  contexts. 
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For  example,  suppose  that  the  Software  Configuration  Management  (SCM)  KPA  was 
investigated  during  an  SCE,  and  that  the  following  observations  were  made  about  the 
processes  in  use  at  a  particular  development  organization  site: 

•  The  investigation  revealed  well-documented  procedures  for  the  SCM  change 
control  process. 

•  The  investigation  noted  that  no  training  was  available  for  software  development 
personnel  in  the  change  control  procedures. 

•  The  investigation  revealed  an  automated  library  system  in  use  (but  only  on  one 
project)  that  supported  and  enforced  the  procedures. 

•  The  investigation  revealed  that  there  was  a  plan  in  place  for  implementing  the 
library  system  across  all  of  the  projects. 

The  findings  for  this  KPA  might  be  that  there  was  a  strength  (the  well-documented 
procedures),  a  weakness  (the  lack  of  available  training),  and  an  improvement  activity  (the 
automated  library  system  and  the  plan  for  implementing  it  across  the  organization). 

The  outcome  would  then  depend  on  the  «*•  use  (or  application)  of  the  SCE  Method.  The 
findings  belong  to  the  sponsoring  organization  and  could  be  used  in  many  different  ways — that 
is,  the  outcome  could  be  different.  For  example,  in  a  government  source  selection,  the  findings 
might  be  factored  into  a  risk  determination.  The  development  organization  might  be  given  a 
“moderate”  risk  rating  for  Software  Configuration  Management  based  on  the  findings.  The 
individual  risk  ratings  for  all  the  KPAs  evaluated  during  an  SCE  would  result  in  a  composite 
SCE  risk  rating.  This  factor  would  be  considered  along  with  many  others  (such  as  cost, 
technical  evaluations,  prior  performance,  etc.)  when  awarding  the  contract.  On  the  other  hand, 
in  a  process  monitoring  situation,  the  same  findings  might  lead  the  sponsoring  organization  to 
require  that  the  automated  library  system  be  implemented  on  their  development  project,  and 
some  portion  of  an  award  fee  might  be  tied  to  successful  implementation  of  a  training  program 
in  the  procedures  for  SCM  change  control.  In  an  internal  evaluation,  the  findings  might  be  used 
to  set  up  an  action  plan  for  implementing  throughout  the  rest  of  the  organization. 

Fundamentally,  the  SCE  data  collection  model  is  used  without  change  in  any  application  of 
the  method.  However,  how  the  team  uses  this  data  collection  model  depends  on  how  the 
method  itself  is  tailored  during  the  Plan  and  Prepare  for  Evaluation  Phase  to  meet  sponsor 
needs.  Recall  the  three  primary  SCE  application  types:  supplier  selection,  process  monitoring, 
and  internal  evaluation.  These  application  types  represent  “families”  of  evaluations  that  share 
similar  characteristics.  The  table  below  illustrates  how  an  SCE  might  differ  from  one 
application  to  the  next  based  on  common  parameters  for  describing  an  evaluation. 
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SCE  Application 

Supplier  Selection 

Process  Monitoring 

Internal  Evaluation 

Motivation 

Obtain  best  value  supplier 

Improve  supplier 
performance 

Improve  organization  or 
project  performance 

Goal 

Risk  identification 

Risk  management 

Risk  reduction  and 
organizational  improvement 

Objective 

Support  decision  making: 
determine  process  risks 

Support  decision  making: 
baseline  and  measure 
progress;  understand  team 
capability 

Support  decision  making: 
baseline  and  measure 
progress;  obtain 
independent  review 

Outcome 

Award  decision:  executable 
contract 

Award  fee  or  value 
engineering  decision 

Revised  action  and 
management  plans 

Competition  readiness 
decision 

Ownership  of 
Results 

Buyer,  may  be  shared  with 
supplier 

Joint,  buyer  and  supplier 
teams 

Supplier  management 

Evaluation 

Emphasis 

Evaluate  capability: 
verify  process, 
validate  information 

Measure  progress 

Check  status: 
process  planning, 
management,  and  control 

Table  2-1: 

Application  Differences 

Various  appraisals  differ  in  their  motivation,  goal,  objective,  outcome,  ownership  of  results, 
and  appraisal  emphasis.  While  there  is  much  overlap  between  types,  key  differences  separate 
them.  A  simple  heuristic  for  thinking  about  what  application  type  is  most  similar  to  your  need 
is  to  ask,  “Who  is  the  sponsor,  who  will  be  on  the  team  to  conduct  the  evaluation,  and  how  will 
the  results  be  used?”  The  answers  to  these  questions  will  principally  determine  the  application 
type.  For  example,  if  the  sponsor  is  a  customer,  and  the  evaluation  will  be  conducted  by  an 
entirely  external  team  (either  a  customer  or  third  party  team),  and  the  results  will  be  used  to 
determine  a  contract  award(s),  you  can  be  sure  that  this  is  a  supplier  selection  application  of 
the  SCE  Method.  In  this  example,  the  motivation  is  to  obtain  the  best  value  supplier,  and  the 
outcome  is  an  award  decision. 

Specific  tailoring  tables  of  characteristics  that  affect  risk  in  the  evaluation  results  are  provided 
in  evaluator  training  (e.g.,  team  size,  time  on  site,  team  composition,  CMM  scope,  etc.). 
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Part  3  Activity  Descriptions 

This  part  of  the  document  contains  the  following  sections: 
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Section 


This  part  describes  the  SCE  Method  in  detail,  with  the  primary  focus  on  what  is  done  rather 
than  how  it  is  done.  The  SCE  V3.0  Method  is  composed  of  three  phases  with  1 5  activities. 

Each  activity  section  follows  a  consistent  structure  for  describing  each  activity: 

•  a  table  summarizing  the  phase,  a  specific  activity,  its  associated  steps,  and  outcome 

•  an  activity  diagram 

•  purpose 

•  inputs 

•  action 

•  outputs 

•  outcome 

•  options  (such  as  tailoring  for  acquisition) 
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Table  3-1  below  summarizes  the  activities  of  the  SCE  method  and  their  purpose. 

1  Phase 

I  Activity 

Purpose  | 

Plan  and 
Prepare  for 
Evaluation 

1. 

Analyze 

Requirements 

Understand  the  business  needs,  objectives,  and  constraints  to  design 
the  most  appropriate  appraisal,  and  to  gain  sponsorship  and 
commitment  for  the  appraisal. 

2. 

Develop 
Evaluation  Plan 

Create  an  artifact  which  will  guide  and  define  execution  of  the 
appraisal  such  that  it  is  in  concert  with  business  needs  and 
constraints. 

3. 

Select  and 
Prepare  Team 

Ensure  that  an  experienced,  trained  team  is  available  and  ready  to 
execute  the  appraisal  process. 

4. 

Obtain 

Organizational 

Information 

Obtain  information  from  the  organization  that  allows  the  team  to  refine 
the  plan  and  focus  the  investigation  in  subsequent  activities. 

5. 

Analyze 

Instrument 

Data 

Identify  issue  areas  for  further  investigation  during  evaluation  conduct, 
and  get  a  preliminary  understanding  of  the  organization’s  operations. 

6. 

Select  and 

Prepare 

Participants 

Ensure  the  most  appropriate  personnel  participate  in  the  evaluation, 
and  ensure  that  they  understand  the  appraisal  process  and  shared 
expectations. 

7. 

Prepare  For 

Data  Collection 

Plan  the  detailed  site  intervention  to  make  optimum  use  of  available 
site  visit  time  to  attain  evaluation  goals  and  objectives. 

Conduct 

Evaluation 

8. 

Receive 

Presentations 

Collect  data  by  allowing  organization  personnel  to  explain  their 
process  (e.g.,  in  presentations). 

9. 

Review 

Documents 

Collect  data  by  examining  process  artifacts  (e.g.,  documents). 

10. 

Conduct 

Interviews 

Collect  data  by  interviewing  process  agents  (e.g.,  managers, 
practitioners,  and  process  owners). 

11. 

Consolidate 

Data 

Transform  the  data  collected  into  formal  team  observations  of  process 
strengths  and  weaknesses  relative  to  the  reference  model  (e.g.,  the 
CMM). 

12. 

Deliver  Draft 
Findings 

Validate  observations  and  collect  data  by  conducting  interactive 
feedback  sessions  with  participants. 

13. 

Make  Rating 
Judgments 

Make  decisions  about  the  organization’s  process  capability,  based  on 
validated  observations,  relative  to  the  reference  model  components 
investigated. 

Report 

Results 

14. 

Deliver  Final 
Findings 

Provide  a  clear  and  actionable  summation  of  the  evaluation  results  to 
the  sponsor  and  the  organization. 

15. 

Produce 

Reports  and 
Support  Follow- 
On  Activities 

Produce  a  formal  baseline  of  the  appraisal  conduct  and  results  for  the 
sponsor  and  other  stakeholders,  and  ensure  the  evaluation  results  are 
used  appropriately  to  achieve  stated  business  objectives. 

Table  3-1:  SCE  V3.0  Activities  and  Their  Primary  Purpose 
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3.1  Activity  1  Analyze  Requirements 

Table  3-2  and  Figure  3-1  provide  an  overview  of  the  steps  in  this  activity. 


Activity 


Steps 


Outcome 


Analyze 

Requirements 


Step  1A:  Determine  Evaluation  Goals 
Step  1 B:  Determine  Evaluation  Constraints 
Step  1C:  Determine  Reference  Model  Scope 
Step  ID:  Determine  Organizational  Scope 
Step  1 E:  Determine  Evaluation  Outputs 
Step  1 F:  Obtain  Commitment 


The  decision  to  proceed  with  the 
evaluation,  based  on  a  shared 
understanding  of  the  evaluation  goals, 
constraints,  and  scope. 


Table  3-2:  Analyze  Requirements 

Purpose  Understand  the  business  needs,  objectives,  and  constraints  in  order  to  design 

the  most  appropriate  appraisal,  and  to  gain  sponsorship  and  commitment  for 
the  evaluation. 


inputs  Evaluation  requirements  which  include  business  context,  sponsor  objectives, 

and  specific  sponsor  requirements 

Resource  constraints  which  include  budget,  personnel,  facility,  and  project 
availability 

Logistical  constraints  which  include  program  plans,  external  schedule, 
geographic  constraints 

Action  This  activity  includes  development  of  evaluation  goals,  constraints,  and  scope. 

Evaluation  outputs  and  their  use  are  determined.  Development  of  the 
evaluation  plan  (Activity  2)  may  begin  in  parallel  with  obtaining  commitment. 
Analyze  Requirements  ends  with  a  commitment  from  the  sponsor  to  proceed 
or  not  to  proceed. 

Determine  revaluation  goals.  The  sponsor  conveys  the  evaluation  goals  to 
the  team  leader.  Evaluation  goals  will  be  directly  correlated  to  the  overall 
business  objectives  of  the  sponsor  and  the  context  within  which  the  evaluation 
is  being  conducted.  Some  typical  goals  for  SCE,  which  are  directly  related  to 
the  specific  application  of  the  method,  include  risk  identification,  risk 
management,  and  risk  reduction  and  organizational  improvement.  Risk 
identification  is  for  the  purpose  of  selecting  the  best  value  supplier  for  a 
contract.  Risk  management  is  for  improving  supplier  performance  during 
execution  of  existing  contracts.  Risk  reduction  is  aligned  with  internal 
evaluations  and  the  need  to  improve  organizational/project  performance  or 
process  capability  to  improve  competitiveness  for  future  business. 
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Note:  The  Plan  and  Prepare  phase  is  a  highly  iterative  set  of  activities.  For  example,  in  this  activity,  outputs  might 
be  determined  early  in  the  activity  and  would  affect  the  reference  model  scope  selected  if  maturity  level  ratings 
were  needed.  The  diagrams  are  principally  a  simple  graphical  representation  of  the  steps  that  must  occur  during 
the  activity,  generally  when  they  occur,  and  generally  what  must  occur  before  subsequent  steps  are  completed. 
Most  of  the  activity  diagrams  are  easier  to  follow  when  beyond  the  Plan  and  Prepare  phase  (Activities  1-7). 


For  example,  if  an  evaluation  is  being  conducted  internally  to  help  prepare  a 
site  for  an  upcoming  customer  led  SCE  (risk  reduction),  it  will  imply  certain 
types  of  activities  for  the  team.  Alternatively,  if  the  sponsor  wants  to  perform 
evaluations  on  existing  subcontractors  as  a  means  of  starting  joint 
improvement  efforts  (risk  management),  that  implies  a  different  set  of  planning 
considerations. 


36  ©  1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University  CMU/SEI-96-TR-002 


April  1996 


Activity  Descriptions 


As  part  of  determining  evaluation  goals,  the  sponsor  and  team  leader  (or 
another  technical  focal  point  if  the  team  leader  has  not  yet  been  selected)  will 
make  the  decision  of  what  type  of  rating  procedure  will  be  done.  This  decision 
is  related  to  evaluation  goals  because  the  rating  baseline  decision  is  a  critical 
method  tailoring  parameter.  This  decision  making  step  is  called  determining 
the  rating  baseline. 

Table  3-2  depicts  the  two  options  available  for  determining  the  »*■  rating 
baseline.  The  chosen  rating  baseline  option  affects  how  much  of  the  reference 
model  is  investigated  (scope)  and  the  amount  of  detail  data  within  the  scope 
that  must  be  collected  during  the  evaluation  (coverage).  Each  option  implies  a 
different  amount  of  "*•  appraisal  risk  that  affects  «+■  method  tailoring 
decisions  made  during  planning  (Activity  2). 


Depth  Breadth 


Partial  scope  /  full  coverage  Full  scope  /  partial  coverage 

Based  on  a  subset  of  model  components  Full  model  scope 

Maturity  level  rating  is  not  feasible  Maturity  level  rating  is  feasible 

Items  rated  only  when  fully  covered 


Table  3-3:  Determining  the  Rating  Baseline  Option 

The  rating  baseline  decision  directly  affects  the  reference  model  scope 
selected.  It  indirectly  affects  the  organizational  scope  and  drives  the  time, 
labor,  and  cost  of  the  evaluation  determined  in  Activity  2,  Develop  Evaluation 
Plan.  For  example,  if  the  sponsor  selects  the  depth-oriented  option,  the  team 
may  be  able  to  completely  cover  the  critical  components  relatively  quickly. 
Alternatively,  a  decision  to  rate  maturity  level  based  on  a  full  scope  and 
complete  coverage  of  the  model  will  require  more  resources  than  the  baseline 
SCE  V3.0  approach. 

The  SCE  V3.0  method  allows  two  options  for  the  rating  baseline.  The  options 
are  called: 

1 .  Depth-oriented  —  partial  scope  (within  a  maturity  level’s 
components),  complete  coverage. 

This  option  is  most  similar  to  SCE  V2.0  principles.  It  is  the  baseline  SCE  V3.0 
method  approach.  Here  a  formal  sampling  technique  is  used  to  select  parts  of 
the  model  for  investigation  that  are  most  similar  to  the  “target”  product  profile, 
or  those  processes  that  will  bear  the  greatest  risk  on  project  success.  Complete 
coverage  is  required  for  those  components  that  are  selected  for  rating.  A 
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maturity  level  rating  is  not  possible  in  this  option  due  to  the  sampling  approach 
used  to  select  model  components.  Key  process  area  ratings  are  not  be 
possible  unless  all  of  the  goals  for  that  KPA  are  rated. 

2.  Breadth-oriented  —  full  scope 
There  are  two  sub-options,  either 

•  obtain  complete  coverage  of  the  model  components  selected,  or 

•  rate  model  components  without  complete  coverage. 

Full  scope  requires  evaluation  of  all  KPAs  within  chosen  maturity  levels. 
Obtaining  complete  coverage  for  a  full  scope  evaluation  is  the  most  time 
consuming  option.  If  the  sub-option  to  rate  without  complete  coverage  is 
chosen,  the  team  must  determine  a  coverage  factor  and  report  the  extent  of 
coverage  to  the  sponsor  along  with  the  rating.  Basically,  this  simply  means  that 
if  the  team  doesn’t  investigate  an  entire  item,  that  information  should  be 
presented  to  the  sponsor  so  that  he  or  she  can  make  the  most  informed 
business  decisions.  Coverage  factors  apply  to  the  practices,  common  features, 
and  goal  components  of  the  reference  model. 

Figure  3-2  depicts  the  rating  baseline  options,  and  includes  a  list  of  some  key 
evaluation  characteristics  if  the  option  is  selected. 


Depth-Oriented 


Full  model  scope  - 
two  sub-options 

•  full  coverage  of  items,  or 

•  coverage  factor  rating 
without  full  coverage 
Maturity  level  rating  is 
feasible  in  both  sub-options 


Selected  model  components 

•  based  on  subset  of  model 
components 

•  items  rated  only  when  fully  covered 

•  decision  to  rate  items  made  in 
planning 

•  maturity  level  rating  is  not  feasible 


Figure  3-2:  Determining  the  Rating  Baseline  Option 
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The  rating  baseline  options  in  SCE  V3.0  are  intended  to  meet  the  following 
objectives,  which  are  elaborated  in  other  parts  of  this  document: 

•  provide  options  to  users  (user  flexibility) 

•  allow  maturity  level  rating  (responsiveness  to  sponsors) 

•  rate  items  that  are  fully  covered  (judgment  principle) 

•  decide  what  to  rate  up  front  (planning  principle) 

The  coverage  factor  ratings  noted  in  the  figure  above  always  go  hand  in  hand  with  the  rating 
values  for  model  components,  and  when  determined,  are  always  part  of  the  final  findings 
briefing  (see  Activity  13,  Make  Rating  Judgments,  and  Activity  14,  Deliver  Final  Findings).  A 
coverage  factor  rating  is  effectively  a  “caveat”  on  the  rating  provided  to  the  sponsor  -  when 
any  sampling  approach  is  used,  there  is  a  chance  that  the  information  collected  may  not 
accurately  reflect  the  state  of  the  entire  set.  In  SCE,  a  coverage  factor  rating  provides  the 
sponsor  with  a  notion  of  how  “complete”  the  team’s  analysis  was,  and  the  sponsor  can  act 
accordingly  in  subsequent  decision  making  related  to  the  appraisal  results. 

When  the  option  is  selected  to  rate  key  practices  and/or  common  features,  they  should  be 
rated  before  the  rating  of  goals  in  the  following  manner: 

Key  practices  should  be  rated  first.  If  there  are  weaknesses  identified  that,  in  aggregate, 
significantly  impact  the  achievement  of  the  KPA  goal(s),  the  key  practice  should  be  rated  as 
not  satisfied,  otherwise  it  should  be  rated  satisfied.  Common  features  should  be  rated  after  the 
key  practices.  If  any  of  the  associated  key  practices  are  rated  unsatisfied,  then  the  common 
feature  is  automatically  rated  unsatisfied.  If  the  associated  key  practices  are  all  rated  satisfied 
or  have  not  been  rated,  then  the  rating  of  the  common  features  is  based  on  whether  or  not 
identified  weaknesses,  in  aggregate,  significantly  impact  the  achievement  of  KPA  goal(s). 

Once  the  key  practices  and/or  common  features  have  been  rated,  the  KPA  goals  can  be  rated. 
If  any  of  the  lower  level  components  have  been  rated  unsatisfied,  then  the  KPA  goal(s)  that 
are  affected  are  rated  unsatisfied.  If  the  lower  level  components  that  map  to  a  particular  goal 
are  all  rated  satisfied,  then  the  rating  of  the  KPA  goal  is  based  on  whether  or  not  the  identified 
weaknesses  that  map  to  the  goal,  in  aggregate,  significantly  affect  the  achievement  of  the 
KPA  goal. 

Determine  evaluation  constraints.  The  purpose  of  this  step  is  for  the  sponsoring 
organization  to  understand  the  attributes  of  the  product  and  establish  resource  boundaries 
within  which  the  team  must  plan  and  execute  the  evaluation.  The  sponsoring  organization 
generates  a  m  profile  of  product  attributes  (the  Product  Profile)  for  the  product  to  be 
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developed.  These  attributes  are  a  constraint  on  the  processes  that  will  be  needed  and  used 
to  create  the  product.  The  attributes  used  in  SCE  are  defined  in  Appendix  E,  and  are  illustrated 
in  the  example  below. 

An  example  Product  Profile  is  shown  in  Table  3-3  below.  The  Product  Profile  should  be 
developed  by  people  with  appropriate  engineering  experience.  If  the  SCE  team  leader  has 
been  selected  (see  Activity  3),  he  or  she  should  help  with  this  effort. 


1  Attribute  Name 

1  Example  Product  Profile  jj 

Application  Domain 

Command  and  control 

Product  Type 

Size 

Anti-Submarine  Warfare  (ASW)  —  helicopter 
sono-buoys 

Duration 

24  months, 

Team  size 

100  software  engineers, 

Estimated  size 

300  Thousand  Source  Lines  of  Code  (KSLOC) 

Reuse  estimate 

30%  new 

35%  modified 

35%  reused 

Large  amount  of  commercial-off-the-shelf  (COTS)  and 
non-developmental  item  (NDI)  integration 

Type  of  Work 

Full  development 

Development  Team  Approach 

Integrated  Product  Teams 

Language(s) 

Ada 

Customer 

Naval  Air  Systems  Command 

Applicable  Standards 

MIL-STD-498 

Major  Subcontractors 

Yes 

Precedence 

Yes  -  replacement  of  existing  system 

Target(s) 

M68000 

Table  3-4:  Example  Product  Profile 

The  profile  is  also  used  to  refine  the  evaluation  scope  to  best  meet  sponsor 
goals  within  identified  constraints.  Conversely,  the  scope  can  be  used  to 
identify  resources  the  sponsor  needs  to  allocate  to  the  evaluation  (see  the 
temporal  flow  in  the  appendices  for  baseline  method  time  estimates).  The 
constraints  established  will  be  input  to  Activity  2,  Develop  Evaluation  Plan,  to 
be  refined  and  finalized.  The  SCE  V3.0  Implementation  Guide  for  Supplier 
Selection  [Barbour  96]  provides  additional  cost  and  schedule  estimates  for  an 
acquisition  application. 
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Determine  CMM  scope.  Factors  to  consider  in  determining  the  CMM  scope 
include  the  following: 

•  rating  baseline  option  chosen 

•  evaluation  constraints  (such  as  the  product  profile,  time,  and  cost) 

•  Target  Process  Capability  selected 

Evaluation  goals  (evidenced  by  the  rating  baseline  option  chosen)  and 
constraints  (evidenced  by  the  product  profile)  are  used  to  define  the  reference 
model  scope.  The  sponsoring  organization  determines  the  key  process  areas 
to  be  evaluated.  These  key  process  areas  form  the  boundary,  or  Target 
Process  Capability,  of  the  evaluation.  An  example  would  be  to  select  all  of  the 
level  two  and  level  three  KPAs  from  the  CMM  for  Software,  VI  .1 .  This  is  a 
typical  TPC  for  many  types  of  organizationally  oriented  evaluations.  A  process 
monitoring  or  internal  evaluation  SCE  might  choose  a  particular  subset  of 
KPAs  based  on  the  results  from  previous  appraisals.  The  process  of  refining 
the  scope,  using  key  process  area  goals,  common  features,  and  practices,  may 
also  begin  during  this  activity  (the  fully  refined  scope  is  completed  during 
Activity  2  and  Activity  7). 

The  purpose  of  this  step  is  to  determine  the  process  capability  that  is  most 
appropriate  for  the  business  goals,  and  to  document  the  desired  capability  as 
the  Target  Process  Capability. 

This  step  should  be  done  by  senior  people  with  engineering  experience  who 
have  a  good  understanding  of  the  Product  Profile  attributes  and  process 
management  concepts.  If  the  SCE  team  members  have  been  selected,  they 
should  help  with  this  effort. 

The  Target  Process  Capability  establishes  the  boundaries  of  the  evaluation  at 
a  high  level.  The  key  process  areas  are  the  basis  for  evaluation  at  the 
development  organization  site(s)  —  a  key  process  area  is  evaluated  if  and  only 
if  it  is  identified  in  the  Target  Process  Capability.  Organizations  must 
understand  what  to  expect  when  an  SCE  is  conducted;  the  Target  Process 
Capability  must  be  communicated  to  the  development  organization(s)  in  order 
to  facilitate  the  on  site  process. 
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"*  Determine  organizational  scope.  Organizational  scope  includes  the 
number  and  type  of  projects  selected  for  investigation.1  The  type  of  projects 
selected  is  directly  related  to  the  product  profile.  This  decision  is  input  into 
planning,  and  will  affect  what  portion  of  the  organization  is  evaluated  (i.e.,  the 
appraised  entity)  (see  Activity  6,  Select  and  Prepare  Participants).  In  this 
activity,  the  organizational  scope  is  defined  at  a  high  level. 

The  selection  of  organizational  scope  and  the  selection  of  reference  model 
scope  are  mutually  reinforcing  steps  that  are  iterative  in  nature.  The  standard 
SCE  for  acquisition  includes  three  projects  within  a  product  line  or  domain  most 
similar  to  the  planned  program.  A  process  monitoring  SCE  focuses  on  one 
project  —  the  one  being  implemented.  The  breadth  of  the  organization  that  is 
decided  to  be  evaluated  will  affect  the  depth  within  the  reference  model  that 
can  be  fully  covered,  given  the  cost  and  time  constraints  that  are  fleshed  out  in 
planning  (Activity  2). 

Determine  evaluation  outputs.  Evaluation  outputs  are  documented  in  the 
findings  briefing  delivered  to  the  organization  and  sponsor  at  the  conclusion  of 
the  site  visit  (see  Activity  14).  The  outputs  always  include  the  findings  of 
strengths  and  weaknesses  in  the  key  process  areas  investigated,  and  usually 
include  the  results  of  rating  judgments  made  by  the  team  (see  Activity  13). 
Understanding  how  the  outputs  will  be  used  is  essential  for  detailed  planning 
(see  Activity  2). 

For  example,  if  the  outputs  (results)  will  be  used  to  determine  the  risks  in 
awarding  a  contract  to  a  supplier,  this  information  will  affect  the  constraints, 
model  scope,  and  organizational  scope.  This  acquisition  example  implies  a 
specific  set  of  attributes  of  the  acquired  product,  a  specification  of  model 
components  to  be  investigated  in  detail,  and  full  coverage  of  the  model 
components  chosen  for  investigation. 

Obtain  commitment.  This  is  the  ultimate  objective  of  Activity  1 .  All  of  the  steps 
involving  direct  consultation  with  the  sponsor  are  geared  not  only  to  determine 
information  critical  to  the  planning  and  conduct  of  the  evaluation,  but  also  to 
ensure  the  sponsor  is  committed  to  the  effort.  It  is  feasible  that  in  analyzing 
requirements,  the  sponsor  may  decide  not  to  go  ahead  with  the  appraisal.  The 
sponsor  will  reemphasize  his  or  her  commitment  throughout  the  process, 
including  speaking  at  the  opening  meeting  (see  Activity  6),  and  participating  in 
the  final  briefing  (see  Activity  14).  Commitment  by  the  sponsor  is  most  visibly 


1  The  organizational  scope  is  finalized  during  Activity  6,  Select  and  Prepare  Participants,  when  the  specific  site, 
projects,  and  evaluation  participants  are  selected.  Organizational  scope  includes  the  site,  location,  projects, 
and  participants. 
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evidenced  in  writing  as  a  sign-off  on  the  evaluation  plan.  The  initial  commitment 
may  be  to  go  forward  with  the  planning  process.  The  final  “sign-off’  may  be 
shortly  before  the  site  visit,  based  on  refinements  made  through  subsequent 
information  gained  or  analyzed  in  other  activities. 


Evaluation  requirements 

•  evaluation  goals 

•  evaluation  constraints 

•  product  profile 


Evaluation  scope 

•  reference  model  scope 

•  organizational  scope 

List  of  evaluation  outputs 
Sponsor  commitment 

The  decision  to  proceed  with  the  evaluation,  based  on  a  shared  understanding 
of  the  goals,  constraints,  and  scope. 

Team  Leader  Selection  (Activity  3A)  may  occur  immediately,  and  this  selection 
may  be  an  input  into  Activity  1 .  This  is  a  preferable  option.  However,  the 
baseline  method  assumes  that  in  most  applications  many  of  the  Activity  1 
decisions  must  be  made  by  the  sponsor  prior  to  Team  Leader  Selection. 

Rating  baseline  options  are  as  indicated  in  the  body  of  this  section. 

Additional  constraints  may  affect  an  SCE  for  Acquisition.  For  example,  the 
sponsor  is  not  the  senior  site  manager.  In  this  case,  approval  for  the  evaluation 
should  be  obtained  not  only  from  the  sponsor,  but  also  through  formal 
correspondence  with  the  senior  site  manager  for  the  appraised  entity. 
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3.2  Activity  2  Develop  Evaluation  Plan 

Table  3-5  and  Figure  3-3  below  provide  an  overview  of  the  steps  in  this  activity. 


1  Activity 

1  Steps 

1  Outcome  1 

Develop 
Evaluation  Plan 

Step  2A:  Identify  Required  Resources 

Step  2B:  Identify  Cost 

Step  2C:  Identify  Schedule 

Step  2D:  Work  Out  Logistics 

Step  2E:  Tailor  Method 

Step  2F:  Plan  Use  Of  Evaluation  Outputs 

The  sponsor  and  team  leader  agree  on  the 
planned  course  of  action  for  the  evaluation. 

Table  3-5:  Develop  Evaluation  Plan 


Purpose  Create  an  artifact  which  will  guide  and  define  execution  of  the  evaluation  such 
that  it  is  in  concert  with  business  needs  and  constraints. 

inputs  Evaluation  goals,  evaluation  constraints,  reference  model  scope, 

organizational  scope,  list  of  evaluation  outputs  (Activity  1),  team  leader 
selection  (Activity  3),  site  information  (Activity  4) 

Action  This  activity  includes  both  development  of  an  initial  plan  and  refinement  of  that 

plan.  It  may  overlap  Activities  3-6.  Planning  for  use  of  evaluation  outputs 
started  in  Activity  1  is  completed  during  this  activity.  The  evaluation  plan  is 
perhaps  the  most  important  artifact.  It  will  guide  and  define  execution  of  the 
appraisal  such  that  it  is  in  concert  with  the  business  needs  and  constraints.  An 
initial  plan  can  be  generated  immediately  following  consultation  with  the 
sponsor.  Further  refinement  is  done  as  detailed  planning  occurs,  and  new 
information  comes  to  light  in  executing  Activities  3-6.  A  final  evaluation  plan 
must  be  completed  prior  to  the  end  of  Activity  7,  Prepare  for  Data  Collection. 
Typically,  resources,  method  tailoring,  and  planning  use  of  outputs  are  finalized 
early  on,  while  cost,  schedule,  and  logistics  are  finalized  later  in  the  Plan  and 
Prepare  for  Evaluation  phase. 

Identify  resources,  cost,  and  schedule.  These  three  items  go  hand  in  hand. 
The  sponsor  will  convey  some  indication  of  these  parameters  during  Analyze 
Requirements  (Activity  1).  In  Activity  2,  these  items  are  refined  and  finalized. 
Trade-offs  are  planned  as  a  routine  course  of  action.  An  example  would  be  as 
follows: 

An  SCE  for  acquisition  will  require  an  in  depth  evaluation  to  assist  risk 
determination  in  contract  award  decision  making.  Five  site  visits  are 
anticipated,  major  subcontractors  teamed  with  primes  are  expected,  and  the 
overall  acquisition  process  must  be  completed  within  three  months.  Given 


44 


©  1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


CMU/SEI-96-TR-002 


April  1996 


Activity  Descriptions 


3A  Select  Team 
Leader 


1 .  Analyze 
Requirements 


2.  Develop 
Evaluation 
Plan 


"Revisions" 
See  Activity 
4 


2A  Identify 

Required 

Resources 


2C  Identify 
Schedule 


2E  Tailor 
Method 


2B  Identify 
Cost 


2D  Work  Out 
Logistics 


2F  Plan  Use 
Of  Evaluation 
Outputs 


4.  Obtain 

Organizational 

Information 


"Revisions" 
See  Activity  4 


Figure  3-3:  Develop  Evaluation  Plan  Activity  Diagram 


these  constraints,  rating  baseline  option  #2  was  chosen  (standard  approach, 
investigating  a  subset  of  the  model  in  depth,  with  full  coverage  but  no  maturity 
level  as  an  output). 

These  decisions  made  in  Activity  1  allow  the  team  leader  to  perform  detailed 
planning.  Completing  the  required  SCEs  will  each  take  the  standard  three 
days.  A  decision  might  be  made  to  charter  two  teams,  running  in  parallel,  one 
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to  evaluate  the  primes  and  one  to  evaluate  the  major  subcontractors.  This 
ensures  fairness,  and  allows  enough  time  per  site  for  the  typical  five  person 
team  to  accomplish  its  mission  within  the  acquisition  schedule. 

Work  out  logistics.  The  team  leader  must  ensure  that  logistical  items  such  as 
facility  planning,  site  locations,  travel  and  hotel  arrangements  are  cared  for. 
Usually,  the  team  leader  will  delegate  some  of  these  tasks  to  a  team  member. 
The  responsible  party  will  work  with  a  site  focal  point,  or  «*■  site  technical 
coordinator,  who  is  identified  to  assist  the  team  throughout  the  process.  In 
many  SCE  applications,  the  site  technical  coordinator  may  not  be  identified 
until  Activity  4,  Obtain  Organizational  Information,  when  the  '«*■  site 
information  packet  is  received. 

Tailor  method.  Method  tailoring  is  directly  related  to  the  organizational  scope 
and  reference  model  scope  steps.  Most  of  the  allowable  tailoring  options  flow 
from  these  decisions.  Tailoring  decisions  always  affect  the  appraisal  risk. 

At  the  highest  level,  the  following  parameters  are  the  most  critical  to  the 
tailoring  decision: 

•  the  purpose  for  conducting  the  evaluation 

•  the  breadth  across  an  organization  probed 

•  the  depth  within  the  reference  model  probed 

•  the  associated  rating  baseline  option  chosen 

•  the  time  spent  on  site, 

•  the  number,  experience,  and  type  of  training  needed  of  the  team  members 
Other  factors  that  affect  tailoring  include: 

•  what  intervention  techniques  are  chosen,  and  in  what  amount  they  will  be 
used  (e.g.,  20%  of  site  time  spent  interviewing  managers,  30%  of  site  time 
spent  interviewing  practitioners) 

•  whether  draft  and  final  findings  will  be  provided  on-site 

•  whether  non-CMM  issues  will  be  evaluated  and  provided 

•  various  templates,  forms,  and  instruments  chosen  (reflecting  the  critical 
tailoring  decisions  above) 

An  example  is  deciding  to  use  fully  automated  support  to  assist  team  members. 
This  would  allow  for  a  large  range  of  data  collection  and  analysis  in  a  broad 
scope  evaluation  that  could  be  processed  in  a  relatively  short  time. 
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Plan  use  of «+  evaluation  outputs.  In  Activity  1,  the  evaluation  outputs  are 
determined.  Specific  planning  for  their  use  is  done  in  Activity  2.  This  is 
particularly  important  in  an  SCE  for  Acquisition,  because  the  standards  for 
analyzing  the  evaluation  results  must  be  specified  during  this  activity.  Planning 
the  use  of  the  outputs  is  the  last  critical  item  which  affects  how  the  data  is 
collected,  and  the  level  of  rigor  for  data  collection  to  support  the  planned  use 
of  the  results. 

Plan  contents.  The  evaluation  plan  should  document  information  on  the 
following  items 

•  evaluation  goals  (including  the  rating  baseline  option)  and  scope  (including 
the  reference  model  and  organizational  scope) 

•  SCE  activities  and  schedule 

•  team  members,  resources,  and  budget 

•  logistical  information 

•  appraised  entity  data  (selected  site,  projects,  participants,  etc.) 

•  specified  tailoring  decisions 

•  planned  use  of  the  evaluation  outputs  and  any  expected  follow  on  activities 

•  known  risks,  and  any  risk  mitigation  or  risk  reduction  activities 

The  final  evaluation  plan  should  have  an  official  “sign-off.”  At  a  minimum,  the 
sponsor  and  the  team  leader  should  sign  the  plan.  The  team  members  and  the 
senior  site  manager,  if  possible,  may  also  sign  off  on  the  plan  to  show 
commitment. 

The  plan  should  be  considered  a  living  document.  Structuring  it  in  a  way  that 
facilitates  easy  update  to  items  that  may  change  during  the  evaluation  (e.g., 
schedule,  interviewees,  etc.)  will  help  keep  it  up  to  date  and  make  it  simpler  to 
obtain  renewed  commitment  from  the  team  and  sponsor,  if  necessary. 

Initial  evaluation  plan,  revised  evaluation  plan 

The  sponsor  and  team  leader  agree  on  the  planned  course  of  action  for  the 
evaluation. 

The  evaluation  plan  does  not  have  to  be  a  single  document.  It  is  preferable  to 
have  one  artifact  which  encapsulates  all  of  the  essential  planning  information. 
However,  because  of  widely  differing  applications  (such  as  a  government 
acquisition  and  a  commercial  internal  evaluation),  this  plan  may  be  the 
collection  of  several  artifacts. 
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In  a  government  acquisition  SCE,  the  plan  will  consist  of  information  contained 
in  several  documents  including  the  acquisition  plan,  the  source  selection  and 
evaluation  plans  (SSP  and  EP),  the  request  for  proposal  (RFP),  and  the  SCE 
team  activity  plan.  The  team’s  briefing  to  prospective  offerors  and  to  the 
appraised  entity  may  also  constitute  documentation  of  the  plan. 
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3.3  Activity  3  Select  and  Prepare  Team 

Table  3-6  and  Figure  3-4  below  provides  an  overview  of  the  steps  in  this 
activity. 


Activity 


Steps 


Outcome 


Select  and 
Prepare  Team 


Step  3A:  Select  Team  Leader  An  experienced,  trained  team  is  ready  to 

Step  3B:  Select  Team  Members  execute  the  evaluation  process. 

Step  3C:  Prepare  Team  (e.g.,  team  training, 
team  orientation,  team  building,  team 
practice) 


Table  3-6:  Select  and  Prepare  Team 


Figure  3-4:  Select  and  Prepare  Team 


Purpose  Ensure  that  an  experienced,  trained  team  is  available  and  ready  to  execute  the 
evaluation  process. 

inputs  Evaluation  constraints,  product  profile,  reference  model  scope,  organizational 

scope  (Activity  1),  evaluation  plan  (Activity  2). 
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This  activity  includes  selection  and  preparation  of  the  evaluation  team.  It  may 
occur  after  obtaining  commitment  (1 F)  and  may  provide  input  into  evaluation 
planning.  Preparing  the  team  usually  includes  team  training,  team  orientation, 
team  building,  and  team  practice. 

The  sponsoring  organization  selects  the  individuals  who  will  conduct  the  SCE. 
All  team  members  must  be  trained.  If  the  sponsoring  organization  selects 
someone  who  is  not  trained,  they  must  schedule  training  and  allow  enough 
time  for  completion  of  training.  Team  selection  should  be  accomplished  by 
someone  senior  enough  in  the  organization  to  commit  the  resources  for  the 
duration  of  the  period  that  SCEs  will  be  performed.  Using  the  team  leader  and 
team  member  selection  guidance  below  will  exceed  all  CAF  requirements. 

The  Product  Profile  and  Target  Process  Capability  help  define  the  expertise  the 
team  needs.  The  team  requires  expertise  in  each  of  the  key  process  areas  in 
the  Target  Process  Capability  and  should  have  expertise  with  the  product  type 
and  application  domain  from  the  Product  Profile. 

Select  team  leader.  The  team  leader  should  be  experienced  both  technically 
and  managerially,  and  have  participated  in  two  or  more  SCEs  as  a  team 
member  prior  to  assuming  team  leadership  duties.  One  team  member  must 
have  at  least  6  years  management  experience.  Preferably,  this  is  the  team 
leader.  Additionally,  the  team  leader  must  have  demonstrated  skills  in 
consulting,  presenting,  and  facilitating.  The  team  leader  should  also  have 
experience  in  managing  teams,  because  executing  the  SCE  Method  is  by 
definition  a  team  centered  activity.  These  are  in  addition  to  meeting  all 
requirements  for  team  members,  and  the  team  member  skills  and  experience 
listed. 

The  team  leader  may  be  selected  at  any  time  in  the  process.  Preferably,  the 
team  leader  is  selected  at  start-up  so  that  he  or  she  may  participate  in 
analyzing  requirements  with  the  sponsor.  Due  to  external  constraints,  such  as 
an  acquisition  schedule  and  procurement  regulations,  the  team  leader  may  not 
be  selected  immediately  at  start-up.  The  team  leader  must  be  selected  before 
initiating  the  planning  process. 

Select  team  members  (team  composition).  At  a  minimum,  the  SCE  teams 
must  have  members  with  an  average  of  10  years  of  appropriate  development 
or  management  experience.  The  team  as  a  whole  must  have  at  least  25  years 
technical  experience  as  well  as  10  years  of  combined  management 
experience.  Experience  in  the  domain  of  method  use  is  also  helpful.  At  least 
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two  team  members  should  have  participated  in  previous  SCEs.  No  team 
member  should  have  less  than  five  years  of  professional  experience.  The  team 
members  are  external  to  the  appraised  entity  (see  options  below). 

The  baseline  SCE  V3.0  Method  calls  for  five  team  members.  This  number  is 
recommended  based  on  evaluation  experience  and  human  dynamics.  Given 
the  average  team  member  experience,  the  typical  evaluation  scope 
investigated,  and  evaluation  constraints,  five  members  is  usually  adequate  to 
ensure 

•  depth  of  experience  in  all  the  necessary  technical  areas  (domain,  reference 
model,  etc.), 

•  evaluation  method  experience 

•  coverage  of  the  evaluation  scope  in  the  allotted  time  and  cost 

•  breadth  to  mitigate  potential  missed  items 

•  ability  to  readily  achieve  consensus 

Teams  under  four  members  may  have  insufficient  depth  and  breadth  of 
experience,  and  those  with  over  six  members  typically  have  problems  reaching 
consensus  within  evaluation  time  constraints. 

Prepare  team.  Preparing  the  team  includes  training  in  the  method  and 
reference  model,  team  building,  and  evaluation  orientation.  For  a  team  to  be 
successful,  several  criteria  must  be  met.  These  criteria  are  discussed  below; 
they  include  training,  team  composition,  team  leadership,  team  member 
experience  and  knowledge,  individual  skills,  and  team  development  skills. 

Training.  All  team  members  must  be  trained  in  the  SCE  Method.  Training 
should  ideally  occur  in  the  two  months  just  prior  to  the  evaluation.  Team 
members  trained  previously  may  require  refresher  training  to  ensure  that  all 
team  members  are  using  the  same  method  baseline.  Additionally,  all  team 
members  need  to  demonstrate  knowledge  of  the  reference  model  used,  either 
through  formal  training  or  other  means  (e.g.,  on  the  job  experience  using  the 
CMM  for  Software).  A  major  part  of  team  building  is  accomplished  through 
training.  If  team  members  were  previously  trained,  time  should  be  allocated  to 
have  the  team  members  rebuild  the  team  through  evaluation  activities. 
Orienting  the  team  on  the  evaluation  plan,  or  participating  in  producing  it,  is  a 
good  mechanism  for  building  a  team  whose  members  were  trained  separately. 
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Team  member  experience  and  knowledge.  The  team  needs  appropriate 
development,  management,  and  acquisition  experience  that  is  applicable  to 
the  business  need,  evaluation  requirements,  and  product  profile.  Collectively, 
the  team  must  have  knowledge  of  and  experience  with 

•  The  application.domain  and  product  type. 

•  The  management  processes  required  to  create  an  effective  environment 
for  the  engineering  and  development  of  a  software  product. 

•  The  major  phases  that  engineering  and  development  of  a  software  product 
must  go  through. 

•  The  support  processes  and  management  environment  required  within  the 
engineering  and  development  of  a  software  product. 

•  The  relationship  between  technology  (in  the  form  of  methods  and  tools) 
and  the  support  processes. 

Individually,  each  SCE  team  member  must  have  the  practiced  skills  to: 

•  Perform  all  the  roles  required  (e.g.,  facilitator,  recorder,  and  participant). 

•  Conduct  interviews  (e.g.,  make  an  interviewee  feel  at  ease,  ask  open- 
ended  yet  focused  questions,  keep  the  interviewee  on  track). 

•  Separate  what  an  interviewee  says  from  what  a  listener  hears  (i.e., 
cognizance  of  differences  between  facts,  inferences,  and  judgments). 

Team  development  skills.  All  of  the  SCE  team  members  must  actively  work  at 
the  initial  team  building  and  then  at  continued  development  of  the  team.  This 
requires  skills  in  consensus  building,  conflict  resolution,  negotiation,  and 
decision  making. 

Another  mechanism  that  has  been  used  by  teams  to  increase  their  skill  level 
and  to  build  team  capability  is  to  perform  a  “practice”  SCE.  Although  this  can 
be  a  resource  intensive  part  of  team  preparation,  the  ability  to  conduct  a 
practice  session  with  a  volunteering  organization  can  provide  significant 
benefits  in  the  learning  curve  of  new  team  members,  and  insight  for  the 
recipient  of  the  form  of  findings  from  the  practice  results. 
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Outputs 


Outcome 

Options 


Team  leader  selection 
Team  member  selection 
Prepared  team 

•  method  training 

•  reference  model  training 

•  team  building 

•  evaluation  orientation 

An  experienced,  trained  team  is  ready  to  execute  the  evaluation  process. 

In  a  reevaluation  or  subsequent  follow  on  evaluations,  some  or  all  of  the  team 
members  may  have  been  trained  at  different  times.  In  this  scenario,  team 
training  is  not  required.  However,  the  tasks  of  team  building  and  team  practice 
become  much  more  important  to  success,  since  the  team  members  may  not 
have  worked  together  before. 

Various  applications  of  the  method  imply  different  team  compositions.  The 
baseline  method  is  for  all  external  team  members.  An  internal,  organizationally 
focused,  improvement  oriented  appraisal  may  necessitate  that  a  majority  of  the 
team  be  from  the  organization  being  assessed,  and  therefore  require  a  tailored 
appraisal,  either  SCE  or  CBA  IPI.  Alternately,  a  process  monitoring  evaluation 
resulting  in  an  customer  focused  action  plan  would  best  be  served  with  a  joint 
customer/supplier  team  composition. 

Different  applications  may  also  require  different  team  sizes.  Although  the 
default  is  five,  with  a  range  of  four  to  six,  this  can  be  tailored.  For  example,  an 
internal  evaluation  to  check  the  status  of  the  configuration  management  KPA 
for  one  project  may  very  well  be  done  with  two  people  who  are  experts  in  that 
area.  The  sponsor  and  team  leader  must  weigh  the  evaluation  requirements 
and  scope  to  select  the  most  appropriate  team  size  and  composition. 
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3.4  Activity  4  Obtain  Organizational  Information 

Table  3-7  and  Figure  3-5  below  provide  an  overview  of  the  steps  in  this  activity. 


1  Activity 

1  Steps 

Outcome 

Obtain 

Organizational 

Information 

Step  4A:  Identify  Information  Needed 

Step  4B:  Request  Information 

Step  4C.  Provide  Information  (Organization) 

Development  organization  information  is 
obtained  by  which  the  team  can  finalize  its 
evaluation  focus,  priorities,  and  logistics. 

Table  3-7:  Obtain  Organizational  Information 


Figure  3-5:  Obtain  Organizational  Information  Activity  Diagram 

Purpose  Obtain  information  from  the  organization  that  allows  the  team  to  refine  the  plan 

and  focus  the  investigation  in  subsequent  activities. 

inputs  Product  profile,  reference  model  scope,  organizational  scope  (Activity  1), 

evaluation  plan  (Activity  2). 
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Action 


Outputs 

Outcome 


Analysis  of  data  collected  during  this  activity  may  result  in  revision  of  the 
evaluation  plan  and  indicate  areas  that  the  team  will  probe.  It  includes 
solicitation  of  site  information  and  organizational/project  responses  to 
questionnaires. 

Identify  information  needed.  Identifying  required  information  includes 
providing  guidance  templates  and  instruments  to  the  appraised  organization. 
Organizational  information  may  include  organizational  questionnaires,  project 
questionnaires/profiles,  site  information  packets  (including  organization 
charts),  and  proposed  project  information  (for  acquisition). 

If  a  site  technical  coordinator  has  not  already  been  identified,  the  sponsor 
should  request  that  this  focal  point  be  assigned  and  identified  to  the  sponsor  in 
the  site  information  packet. 

The  questionnaires  used  will  depend  on  the  reference  model  used.  The  CMM 
Maturity  Questionnaire  VI  .1  is  used  in  CMM-based  appraisals.  Additional 
items  such  as  organizational  and  project  questionnaires  may  be  used.  Often, 
they  perform  a  function  similar  to  the  profile  templates.  All  profiles  use  the 
same  attributes  shown  in  the  example  in  Activity  1 .  The  difference  is  in  who  fills 
it  out  and  from  what  perspective  (desired  product,  planned  product,  or  current 
products). 

Request  information.  The  sponsor  formally  requests  the  organization  to 
supply  needed  information  to  the  team.  The  collection  of  items  provided  to  the 
team  is  called  the  site  information  packet.  Besides  the  instruments  and  profiles 
filled  out,  other  items  requested  and  included  in  the  site  information  packet  are 
current  organizational  charts,  site  terminology,  and  high-level  process 
information,  such  as  a  process  documentation  list. 

Provide  information  (organization).  This  step  is  performed  by  the 
organization  that  will  be  evaluated.  It  is  included  in  the  method  to  ensure 
consistency  in  activities  —  the  team  needs  to  receive  data  from  the 
organization  in  order  to  execute  subsequent  planning  and  preparation 
activities. 

Request  for  organization  information,  site  information  (from  organization). 

Development  organization  information  is  obtained  by  which  the  team  can 
finalize  its  evaluation  focus,  priorities,  and  logistics. 
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Options 


In  some  applications  it  may  be  beneficial  to  the  sponsor  to  have  the  team 
evaluate  the  proposed  project  (one  that  does  not  yet  have  a  track  record  of 
implementation).  In  this  case,  proposed  project  profiles  and  related  data  are 
requested  from  the  organization  in  addition  to  the  other  items  in  this  activity. 
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3.5  Activity  5  Analyze  Instrument  Data 

Table  3-8  and  Figure  3-6  below  provide  an  overview  of  the  steps  in  this  activity. 


Activity 

Steps 

Outcome 

Analyze 
Instrument  Data 

Step  5A:  Receive  and  Summarize  Instrument 
Data 

Step  5B:  Examine  And  Review  Instrument 
Data 

Step  5C:  Develop  Profile(s)  Analyses 

The  team  has  a  high  level  understanding  of 
the  site’s  operations. 

Table  3-8:  Analyze  Instrument  Data 


Figure  3-6:  Analyze  Instrument  Data  Activity  Diagram 


Purpose  Identify  issue  areas  for  further  investigation  during  evaluation  conduct,  and  to 
get  a  preliminary  understanding  of  the  organization’s  operations. 

inputs  Product  profile  (from  Activity  1),  site  information  (from  Activity  4). 

Action  This  activity  includes  review  and  analysis  of  all  instrument  responses.  It  begins 

when  the  first  response  is  received  and  ends  when  all  have  been  obtained  and 
analyzed. 
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Receive  and  summarize  information.  In  the  first  step  the  team  summarizes 
the  information  sent  by  the  organization  in  the  site  information  packet. 
Response  Summary  Sheets  (RSSs)  from  the  maturity  questionnaires  are 
typical  outputs,  since  they  help  the  team  see  patterns  in  the  responses  for 
investigation  on  site. 

Examine  and  review  instrument  data.  This  step  includes  reviewing  all 
instruments,  including  product  profiles  and  questionnaires.  It  can  be  done  in 
parallel  with  analysis  of  the  data. 

«►  Product  Profiles.  Several  profiles  are  analyzed  in  this  activity.  Essentially, 
they  revolve  around  two  perspectives:  the  sponsor  (customer)  view  of  the 
product,  and  the  producer  (developer)  view  of  the  product  (via  examples  of 
current  products). 

•  The  Product  Profile  (planned  or  desired)  from  Activity  1  represents  the 
sponsor  perspective. 

•  The  Product  Profile  (proposed)  from  the  development  organization  (Activity 
4)  represent  the  producer  perspective. 

•  A  Product  Profile  (actual,  current)  for  each  of  the  projects  that  has  been 
submitted  for  evaluation  by  the  development  organization. 

The  development  organization  submits  a  Product  Profile  for  each  project  that 
is  a  candidate  for  evaluation.  The  producer  Product  Profiles  are  based  on  the 
same  attributes  as  the  customer’s  Product  Profile.  The  Product  Profiles  capture 
information  about  products  the  development  organization  has  already 
developed  or  is  developing,  and  they  indicate  development  experience 
pertinent  to  the  “target”  product. 

The  SCE  team  uses  the  profiles  to  identify  areas  in  which  the  development 
organization(s)  may  lack  experience  by  examining  the  attributes  in  the  various 
profiles  submitted. 

Questionnaires.  The  CMM  VI. 1  Maturity  Questionnaire  is  the  most  common 
questionnaire  based  instrument  used  in  an  SCE.  There  are  several  that  can  be 
used 

•  maturity  questionnaire 

•  project  questionnaire 

•  organization  questionnaire 
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The  team  uses  these  instruments  to  better  understand  the  basic  structure, 
environment,  products,  and  processes  in  an  organization.  During  this  step,  the 
team  tabulates  and  organizes  the  data  in  a  way  that  facilitates  analysis  in  the 
subsequent  step. 

Develop  profile  analyses.  This  includes  identifying  attribute  mismatches  and 
inconsistencies  and  anomalies  in  questionnaire  data. 

a.  Identifying  mismatched  attributes.  Mismatched  attributes  are  identified  by 
comparing  the  attributes  in  the  sponsor  (or  target)  Product  Profile  to  the 
Product  Profiles  describing  current  projects2.  The  purpose  of  comparing  is  to 
look  for  similarities,  not  for  exact  matches.  For  example,  a  98  KSLOC  system 
would  be  considered  a  match  for  a  105  KSLOC  system.  Judgment  and  team 
consensus  are  used  to  resolve  any  questionable  comparisons. 

An  overall  attribute  mismatch  for  an  organization  is  defined  to  exist  only  if  none 
of  the  Product  Profiles  match  the  sponsor  Product  Profile  for  that  attribute.  A 
Mismatch  Identification  Table  is  created  to  consolidate  the  information 
resulting  from  comparing  the  profiles  submitted  by  a  development 
organization.  Later  in  the  process,  it  may  be  important  to  note  the  relative 
amount  of  experience  in  an  attribute  related  to  the  target  product. 


Project 

Project 

Project 

Project 

Project 

Project 

Org  “A” 

Attributes 

Able 

Baker 

Charley 

Delta 

Foxtrot 

Gamma 

Result 

Application  Domain 

X 

X 

X 

X 

X 

Size 

X 

X 

X 

X 

X 

X 

X 

Table  3-9:  Example  Mismatch  Identification  for  Two  Attributes  for  Organization  A 


Mismatches  for  the  projects  are  shown  by  an  “x”;  matches  are  left  blank.  The  Product  Profile  attributes  for 
Application  Domain  is  acoustic  signal  processing,  and  the  “Estimated  Software  Size”  part  of  the  Size  attribute  is 
1 ,000  KSLOC.  Every  project  submitted  (except  Charley)  was  a  command  and  control  system,  and  the  size  of  each 
was  under  300  KSLOC.  Because  all  projects  submitted  showed  a  mismatch  against  the  sponsor  product  profile  in 
the  Size  row,  the  result  column  is  filled  in  to  indicate  an  overall  mismatch  for  Organization  A.  Also  of  note  is  the  fact 
that  although  only  one  project  had  relevant  experience  in  the  application  domain,  this  is  not  an  overall  mismatch  for 
the  organization. 


2  In  an  SCE  for  Acquisition,  the  development  organization  profiles  are  compared  against  their  proposed  product 
profile.  The  mismatches  looked  for  are  their  current  experience  relative  to  what  they  say  they  will  implement 
on  the  next  project.  The  sponsor  profile  is  used  in  this  case  only  to  help  illuminate  major  gaps  in  technical 
understanding  of  the  problem,  indicated  by  mismatches  between  the  developer’s  proposed  product  profile  and 
the  sponsor’s  product  profile.  This  is  important  because  in  acquisition  the  sponsor  will  always  be  different  from 
and  external  to  the  SCE  recipient  organization. 
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Outputs 

Outcome 

Options 


The  purpose  of  this  step  is  for  the  SCE  team  to  identify  areas  in  which  the 
development  organization(s)  lacks  experience,  indicating  a  potential  for  risk.  A 
development  organization  must  have  well-defined  processes  to  mitigate  the 
risk,  especially  if  the  development  organization  lacks  product  type  or 
application  domain  experience. 

The  Product  Profile  (from  Activity  1)  represents  a  “customer  view”  of  the 
product  to  be  built,  and  the  Product  Profiles  (from  Activity  4)  represent  a 
“developer  view.”  Both  of  these  give  insight  into  development  process  risk.  If 
there  is  a  close  match  between  the  planned  product  and  the  development 
organization’s  actual  products,  then  the  actual  development  processes 
currently  in  use  are  good  indicators  of  the  processes  that  will  be  used  for  the 
new  development. 

b.  Questionnaire  analysis.  The  team  reviews  the  completed  questionnaire(s) 
from  the  development  organization(s)  to  identify  inconsistencies  and 
anomalies  in  the  responses. 

An  "*•  inconsistency  is  an  apparently  contradictory  response  from  the  same 
project  to  two  (or  more)  questions  on  the  questionnaire  that  relate  to  the  same 
key  process  area.  An  «+  anomaly  is  a  contradictory  response  to  the  same 
question  by  two  (or  more)  projects.  Both  types  of  responses  may  indicate 
issues  that  should  be  probed  on  site.  The  information  is  captured  in  a  form  that 
can  be  used  to  help  prioritize  the  amount  of  time  spent  investigating  each  topic 
area  (Activity  7). 

The  team’s  goal  is  not  to  validate  the  development  organization’s  response  to 
the  questionnaires;  rather,  the  goal  is  to  investigate  the  related  topic  areas  to 
identify  strengths,  weaknesses,  and  improvement  activities.  Annotations  in  the 
comments  column  of  the  questionnaire(s)  often  indicate  what  documentation 
exists  to  support  the  answers  to  the  questions.  This  information  can  be  used 
by  the  team  to  tailor  the  requests  for  documentation  (Activity  7)  to  be  reviewed 
during  the  initial  document  review  (Activity  9). 

Profile  and  questionnaire  analyses. 

The  team  has  a  high-level  understanding  of  the  site’s  operations. 

A  Product  Profile  for  the  proposed  effort  will  also  be  submitted  by  the 
development  organization(s)  and  analyzed  by  the  user  in  acquisition 
applications  of  SCE.  This  Product  Profile  describes  the  development 
organization’s  view  of  the  proposed  effort. 
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One  of  the  tasks  performed  is  checking  if  the  development  organization’s  view 
of  the  product  to  be  built  is  similar  to  the  sponsoring  organization’s  view.  This 
is  done  by  comparing  the  sponsor  Product  Profile  to  the  developer  Proposed 
Product  Profile.  Usually  these  are  nearly  identical.  If  they  differ  greatly,  it  should 
be  investigated  because  it  indicates  a  major  difference  in  understanding  about 
what  the  development  project  entails.  Resolving  these  differences  in 
understanding  is  not  part  of  the  SCE  investigation,  but  should  be  brought  to  the 
attention  of  the  sponsoring  organization. 
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3.6  Activity  6  Select  and  Prepare  Participants 

Table  3-10  and  Figure  3-7  below  provide  an  overview  of  the  steps  in  this 
activity. 


Activity  I  Steps 


Outcome 


Select  and  Step  6A:  Select  Site(s) 

Prepare  Step  6B:  Select  Project(s) 

Participants  Step  6C:  Select  Participants 

Step  6D:  Prepare  Initial  Briefing(s) 
Step  6E:  Conduct  Initial  Briefing(s) 


Site  participants  understand  the  evaluation 
process  and  are  ready  to  take  part. 


Table  3-10:  Select  and  Prepare  Participants 


Figure  3-7:  Select  and  Prepare  Participants  Activity  Diagram 


Purpose  Ensure  the  most  appropriate  personnel  participate  in  the  evaluation,  and 

ensure  that  they  understand  the  evaluation  process  and  shared  expectations. 
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Evaluation  plan,  site  information,  profile  and  questionnaire  analyses. 

This  activity  is  an  important  mechanism  for  obtaining  buy-in  for  both  the 
evaluation  process  and  its  results.  This  activity  enacts  the  first  part  of  the 
classic  communication  technique,  “tell  them  what  you’re  going  to  do,  do  it,  and 
then  tell  them  what  you  did.” 

This  activity  identifies  the  specific  organizational  site(s),  project(s)  and 
participants  in  the  evaluation.  Selecting  the  site,  projects,  and  participants  is 
mutually  reinforcing.  A  schedule  and  set  of  briefings  describing  the  evaluation 
activities  will  be  prepared  and  delivered. 

Some  of  the  factors  to  consider  in  accomplishing  this  activity  are: 

•  management  structure 

•  shared  processes 

•  organizational  and  project  responsibilities 

•  geographic  dispersion  of  the  appraised  entity 

These  factors  will  affect  how  the  team  conducts  the  site  visit,  and  more 
importantly,  how  the  team  conducts  the  site  visit(s)  within  defined  evaluation 
constraints.  Information  to  perform  this  activity  comes  from  the  site  information 
packet  obtained  in  Activity  4. 

Select  site(s).  The  team  needs  to  identify  the  place  where  the  site  visit  will 
occur.  Often,  the  terms  “organization,”  “site,”  and  “appraised  entity”  are  used  to 
discuss  the  same  thing.  However,  in  a  typical  evaluation,  it  is  rare  that  all  three 
terms  reflect  the  same  scope.  The  appraised  entity  is  actually  the  result  of 
selecting  the  organization,  site  and  projects  —  it  is  the  organizational  units  to 
which  evaluation  outputs  apply.  Given  the  site  information  provided  by  the 
organization  in  Activity  4,  the  team  will  select  the  location(s)  where  the  majority 
of  or  most  critical  items  will  be  produced.  For  an  internal  evaluation,  this  step 
is  less  complex,  because  much  more  information  is  known  to  the  team  earlier 
in  the  process.  Remember  that  the  sponsor  and  the  senior  site  manager  may 
not  be  the  same  individual.  In  an  acquisition  SCE,  they  will  always  be  different. 

Select  project(s).The  SCE  team  selects  projects  for  investigation  whose 
attributes  most  closely  match  their  Product  Profile.  The  number  of  projects 
selected  varies  depending  on  the  specific  application  and  the  evaluation  goals 
and  constraints.  The  team  uses  all  of  the  available  information  about  the 
projects  to  make  the  selection.  The  purpose  of  this  step  is  to  select  projects  for 
evaluation  that  give  the  most  insight  into  the  processes  that  will  be  used  on  the 
planned,  or  target,  development  project.  By  evaluating  the  actual  processes 


CMU/SEI-96-TR-002 


©  1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


67 


Activity  Descriptions 


April  1996 


used  on  similar  projects,  the  team  obtains  a  clearer  picture  of  the  processes 
that  are  likely  to  be  used  on  the  planned  development,  or  which  need  to 
improve  to  meet  an  intended  target  capability. 

Select  participants.  The  team  also  needs  to  determine  who  will  participate  in 
the  evaluation.  This  decision,  like  most  everything  in  the  evaluation  process, 
will  depend  on  the  evaluation  goals  and  constraints.  In  all  cases,  participants 
will  come  from  the  projects  selected  and  the  groups  that  support  those  projects. 
Examples  on  how  this  foundation  is  tailored  follow. 

In  an  organizationally  focused  evaluation,  results  are  being  used  to  baseline 
process  capability  across  the  organization.  The  people  selected  to  participate 
in  group  interviews  (see  Activity  10)  might  also  come  from  projects  other  than 
the  ones  selected  for  depth  coverage. 

In  a  highly  focused  process  audit  on  one  project,  the  participants  might  be 
limited  only  to  those  working  directly  on  the  project,  because  an  evaluation  goal 
is  to  measure  progress  against  an  earlier  baseline  in  key  process  areas  under 
the  direct  control  of  that  project. 

A  typical  product  line-oriented,  customer-led  SCE  for  acquisition  will  include 
approximately  25-50  people  in  interviews  and  presentations.  An 
organizationally-focused  internal  evaluation  used  for  appraising  improvement 
progress  may  include  as  many  as  75  people  directly  in  interviews.  Many  more 
can  be  “touched”  through  use  of  instruments  such  as  the  CMM  Maturity 
Questionnaire.  Trade-offs  in  time  and  cost  need  to  be  considered. 

The  participants  selected  must  be  representative  of  the  appraised  entity. 
Factors  to  consider  in  ensuring  this  representation  are: 

•  the  size  of  the  appraised  entity 

•  the  breadth  of  the  appraised  entity 

•  the  characteristics  of  the  appraised  entity  population 

•  the  appraised  entity’s  development  activities 

Clearly,  the  smaller  the  entity,  the  more  likely  a  greater  percentage  of  the 
population  can  participate,  and  reduce  the  risk  due  to  organizational  scope. 
Participants  will  come  from  the  projects  selected,  but  support  group  people 
must  also  be  included  (such  as  from  a  software  engineering  process  group). 
The  participants  should  reflect  the  type  of  people  and  skill  level  that  is  typical 
for  the  appraised  entity.  The  participants  must  have  sufficient  knowledge  of  the 
process  activities  in  the  organization  that  will  be  evaluated  in  the  evaluation 
scope. 


68 


©  1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


CMU/SEI-96-TR-002 


April  1996 


Activity  Descriptions 


Outputs 

Outcome 

Options 


Prepare  and  conduct  initial  briefing(s).  The  team  also  prepares  an  entry 
briefing;  the  entry  briefing  is  used  to  set  the  development  organization’s 
expectations  for  the  site  visit.  Initial  briefing(s)  used  to  prepare  participants  may 
be  conducted  at  various  times.  They  may  begin  shortly  after  the  decision  to 
conduct  the  evaluation,  and  may  be  held  multiple  times  with  different  groups  up 
to  and  including  the  first  day  of  the  site  visit.  Internal  evaluations  may  include 
separate  participants  and  opening  “kick-off”  briefings.  The  objective  is  to 
ensure  participants  know  what  the  evaluation  process  is,  and  why  it  is  being 
done.  Prepared  participants  are  much  more  at  ease,  which  results  in  improved 
data  volume,  quality,  and  communication.  It  makes  the  job  of  the  team  easier. 

Selected  site(s),  selected  project(s),  selected  interviewees,  initial  briefing, 
prepared  participants. 

Site  participants  understand  the  evaluation  process  and  are  ready  to  take  part. 

The  participant  briefing  and  the  opening  meeting  briefing  (evaluation  kick-off) 
may  both  be  delivered  on  the  first  morning  of  the  site  visit.  This  option  may  be 
necessary  when  resources  (financial  or  physical)  are  tightly  constrained,  or 
when  the  team  is  entirely  third  party  and  does  not  interact  directly  with  the 
participants  until  day  one  of  the  site  visit.  Users  should  understand  that  the  risk 
of  participant  resistance  may  increase  in  this  scenario  due  to  lack  of  time  for 
internalizing  the  process,  its  purpose,  and  the  expectations  of  management. 
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3.7  Activity  7  Prepare  for  Data  Collection 

Tabie  3-1 1  and  Figure  3-8  below  provide  an  overview  of  the  steps  in  this 
activity. 


1  Activity 

1  Steps 

1  Outcome  I 

Prepare  for 

Data  Collection 

Step  7A:  Prioritize  Focus  Areas 

Step  7B:  Establish  Interview  Strategy 

Step  7C:  Script  Questions 

Step  7D:  Establish  Document  Review 

Strategy 

Step  7E:  Refine  Evaluation  Team  Roles  And 
Responsibilities 

The  team  has  finalized  all  plans  and 
logistics  and  is  ready  to  conduct  the  site 
visit. 

Table  3-1 1 :  Prepare  for  Data  Collection 

Figure  3-8:  Prepare  for  Data  Collection  Activity  Diagram 
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Activity  Descriptions 


Purpose 


Inputs 


Action 


Plan  the  detailed  site  intervention  to  make  optimum  use  of  available  site  visit 
time  to  attain  evaluation  goals  and  objectives. 

Evaluation  plan,  organization  charts  (from  site  information),  profile  and 
questionnaire  analyses,  selected  site(s),  selected  project(s),  selected 
interviewees. 

This  activity  provides  the  foundation  for  execution  of  subsequent  activities. 
Specific  prioritization  of  desired  information  and  strategies  for  data  collection 
(presentations,  interview,  document  review)  are  developed.  Refinement  of 
roles  and  responsibilities  (e.g.,  librarian,  monitors  or  mini-teams)  is 
accomplished.  Activity  7  is  accomplished  iteratively  with  Activity  5,  Analyze 
Instrument  Data,  and  Activity  6,  Select  and  Prepare  Participants. 

Prioritize  focus  areas  —  the  topic  areas  to  be  investigated.  The  SCE  team 
uses  all  of  the  information  available  to  determine  the  topic  areas  that  will  be 
investigated.  The  team  collectively  describes  and  chooses  their  topic  areas  by 
doing  this  step.  The  topic  areas  refine  the  scope  already  identified  in  Activity  1 , 
and  reflect  the  rating  baseline  decision  made  in  Activity  1 .  The  topics  are  used 
to  help  prioritize  site  visit  time  and  team  focus. 

•  reference  model  topic  areas  are  limited  by  the  boundaries  of  the  Target 
Process  Capability  identified  in  Activity  1 .  At  least  one  topic  area  must  be 
selected  for  each  key  process  area  in  the  Target  Process  Capability. 

•  The  SCE  team  may  select  additional  topic  areas  within  the  Target  Process 
Capability  based  on  their  experience  and  judgment. 

As  noted  above,  team  judgment  is  used  to  select  topic  areas.  All  of  the 
information  available  to  the  team  is  used  to  make  these  judgments.  Factors 
that  might  be  considered  include 

•  mismatches  in  profile  attributes 

•  various  organizational  structures  (For  example,  an  organization  without  a 
separate  quality  assurance  function  might  cause  a  team  to  focus  more  on 
the  verifying  implementation  common  feature,  audits  feature) 

•  a  sponsor’s  “hot”  issues  (For  example,  if  configuration  management  is  a 
known  issue  area  for  a  particular  program  manager  or  for  a  product  line,  the 
team  might  focus  on  the  entire  set  of  topics  available  in  the  configuration 
management  key  process  area,  and  focus  on  fewer  topics  in  other  key 
process  areas.) 

•  resource  constraints  (For  example,  a  team  might  decide  to  focus  on  the 
implementation  topic  areas  and  less  on  institutionalization  topic  areas  due 
to  time  constraints.) 
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•  purpose  of  the  evaluation  (For  example,  if  an  external  sponsor  wants  to 
focus  an  organization’s  attention  on  providing  resources  or  training  to 
specific  activities,  the  team  might  choose  topic  areas  using  the  ability  to 
perform  common  feature,  resources  and  training  features.) 

Once  the  topic  areas  are  selected,  the  scope  of  the  SCE  is  refined  by 
accounting  for  all  evaluation  constraints  and  information  provided  by  the 
development  organization. 

Each  topic  focuses  on  one  feature  of  the  reference  model.  "+■  Features  are 
characteristics  common  to  all  mature  processes;  the  CMM  VI  .1  has  five 
common  features.  The  common  features  and  their  respective  features  are: 

Commitment  to  Perform 

•  leadership  —  providing  adequate  senior  management  sponsorship  and 
senior  management  oversight  of  process  activities. 

•  organizational  policies  —  written  and  communicated  policies  governing  the 
key  process  area. 

Ability  to  Perform 

•  resources  —  providing  adequate  resources  (e.g.,  staff,  funds,  facilities, 
tools,  input  data). 

•  organizational  structure  —  establishing  process  responsibilities  and  the 
organizational  units  to  address  process  activities. 

•  training  —  providing  training  for  groups  and  individuals  (availability  of 
training  and  orientation,  and  its  timeliness,  for  the  people  who  carry  out  the 
process  activities). 

Activities  Performed 

•  plans  and  procedures  —  plans  and  procedures  to  support  the  activity. 

•  work  performed —  the  evidence  of  the  use  of  plans,  procedures,  and 
standards  in  implementing  activities. 

•  tracking —  how  the  work  is  tracked  and  how  problems  are  identified. 

•  corrective  actions  —  the  identification  and  resolution  of  problems. 
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Measurement  and  Analysis 

•  take  measurements  —  taking  measurements  to  determine  the  status  of 
process  activities. 

•  analyze  measurements  —  taking  and  analyzing  measurement  data  to 
determine  the  quality/functionality  of  the  outputs  of  activities. 


Verifying  Implementation 

•  reviews  —  periodic  and  event  driven  reviews  with  senior  and  project 
management. 

•  audits  —  actions  undertaken  to  perform  independent  reviews/audits  of 
process  activities  and  work  products  and  reporting  results. 

Features  indicate  whether  the  implementation  and  institutionalization  of  a  key 
process  area  is  effective,  repeatable,  and  lasting  [Paulk  93a].  Recall  that  a 
feature  as  applied  to  a  key  process  area  goal  constitutes  a  topic  area  for 
investigation.  The  key  process  area  matrix  is  used  to  help  visualize  the 
selection  process.  The  topics  are  used  to  plan  the  preliminary  interview 
strategy  and  develop  an  interview  schedule.  The  schedule  is  closely 
coordinated  with  the  development  organization’s  site  technical  coordinator. 

Topic  areas  are  necessary  because  a  goal  is  too  broad  to  be  directly 
observable;  each  topic  defines  areas  of  observable,  implemented  work 
practices  and  their  degree  of  institutionalization.  Topics  help  the  SCE  team  to 
structure  the  investigation,  providing  consistency  across  key  process  areas. 
Each  topic  serves  as  the  basis  of  a  set  of  questions  that  the  team  can  ask  or 
can  seek  in  document  review.  Topic  selection  is  a  critical  activity;  the  team 
must  balance  adequate  reference  model  coverage  against  coverage  of  the 
organizational  scope,  and  do  so  within  the  evaluation  constraints  and  in  line 
with  the  evaluation  goals. 

Establish  the  interview  strategy  and  scripting  questions.  The  team 
develops  a  high-level  interview  strategy  and  prepares  materials  to  guide  them 
during  the  interviews.  This  activity  has  four  major  components:  (1)  allocating 
time  to  various  data  collection  techniques,  (2)  selecting  interviewees,  (3) 
creating  interview  worksheets,  and  (4)  coordinating  the  interview  schedule. 

•  Allocating  time  to  various  data  collection  techniques.  Based  on  the 
topics  selected,  the  team  estimates  the  amount  of  time  needed  for 
interviewing  and  document  review  given  the  site  visit  time  constraints 
established  in  the  planning  activity. 
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•  Selecting  interviewees.  The  topics  and  the  information  about  the 
organization’s  structure  (from  the  site  information  packet)  is  used  to  decide 
who  will  be  interviewed  about  each  topic.  Interviewees  are  not  selected  as 
individuals,  but  instead  by  position  in  the  organization  or  by  their  functional 
area  (e.g.,  Configuration  Management,  Software  Quality  Assurance, 
project  manager). 

•  Creating  interview  worksheets  and  scripts.  Interview  Worksheets  are 
prepared  with  an  initial  set  of  scripted  questions,  derived  from  the  topics, 
for  each  interviewee;  this  keeps  the  interview  focused. 

•  Coordinating  the  interview  schedule.  Finally,  the  team  decides  the 
preferred  order  for  the  interviews  and  coordinates  with  the  site  technical 
coordinator. 

The  team  must  also  finalize  logistical  items  planned  for  in  Activity  2  —  access 
to  the  facility,  adequate  working  space,  a  conference  room,  telephone  and 
copier  access,  and  so  on.  The  documents  for  initial  document  review  were 
specified,  but  the  team  should  ensure  that  the  documents  will  be  available  in 
the  working  space  assigned  to  them. 

Establish  document  review  strategy.  During  this  step  the  SCE  team 
identifies  the  documents  they  expect  to  review,  especially  for  use  during  the 
initial  document  review.  Once  the  projects  are  selected,  the  SCE  team 
requests  documents  for  the  initial  document  review  from  the  development 
organization.  Document  names  will  vary  from  organization  to  organization,  but 
preliminary  identification  of  documentation  is  critical.  Typically,  the  team 
requests  copies  of  pertinent  organizational  **■  policies,  «►  standards, 
"•^procedures,  and  «*•  directives  relating  to  software  development.  The  team 
also  requests  project-level  procedures,  standards,  and  directives  for  the 
projects  selected  for  review.  This  documentation  defines  both  the  organization- 
level  processes  and  the  high-level  processes  used  on  the  selected  projects. 
The  team  will  also  discuss  how  they  will  divide  the  large  review  task,  establish 
team  norms  for  use  of  documents,  etc. 

Refine  team  roles  and  responsibilities.  This  step  is  the  last  item  prior  to 
starting  the  Conduct  Evaluation  phase.  Roles  and  responsibilities  are 
established  early  on  in  the  planning  and  team  selection/preparation  activities 
(Activities  2  and  3).  Throughout  the  rest  of  the  Plan  and  Prepare  for  Evaluation 
phase,  the  team  will  refine  these  roles  as  they  become  accustomed  to  each 
others’  work  habits  and  skills.  If  not  already  completed,  the  following  roles,  with 
associated  team  norms  and  responsibilities,  should  be  formally  established  so 
that  the  team  “hits  the  ground  running”  at  the  site. 

•  team  leader 

•  key  process  area  mini-teams  or  monitors 
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Outputs 


Outcome 


Options 


•  librarian  —  data  manager 

•  appraisal  process  monitor 

Data  collection  strategy 

•  interview 

•  document  review 

Data  collection  tactics 

•  interview  questions 

•  initial  document  request  list 

•  roles  and  responsibilities 

The  team  has  finalized  all  plans  and  logistics  and  is  ready  to  conduct  the  site 
visit. 

Additional  steps  may  be  necessary  in  some  applications  like  government 
source  selection  to  ensure  the  sponsor  that  a  “level  playing  field”  is  in  place 
prior  to  the  site  visits.  [See  SCE  V3.0  Implementation  Guide  for  Supplier 
Selection .]  [Barbour  96] 

The  quantity  of  topic  areas  selected  as  part  of  prioritizing  site  time  must  reflect 
the  rating  baseline  decision  made  in  Activity  1.  The  standard  SCE  approach  is 
to  select  a  subset  of  model  components  for  investigation  that  best  meet  the 
intent  of  the  sponsor’s  use  of  the  appraisal  results. 

Alternatively,  a  full  scope,  full  coverage  evaluation  would  effectively  mean  that 
all  topic  areas  within  the  key  process  areas  selected  for  the  Target  Process 
Capability  must  be  evaluated  (e.g.,  all  goals  and  all  features  within  a  KPA  for 
the  CMM  for  Software) 
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3.8  Activity  8  Receive  Presentations 

Table  3-12  and  Figure  3-9  below  provide  an  overview  of  the  steps  in  this 
activity. 


Activity  I  Steps  Outcome 


Receive  Step  8A:  Deliver  Presentation  (s)  The  evaluation  team  has  a  refined/updated 

Presentations  (Organization)  understanding  of  the  organization’s 

Step  8B:  Listen  process  operations. 

Step  8C:  Ask  Questions 
Step  8D:  Take  And  Tag  Notes 

Table  3-12:  Receive  Presentations 


Figure  3-9:  Receive  Presentations  Activity  Diagram 
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Purpose 


Inputs 


Action 


Collect  data  by  allowing  organization  personnel  to  explain  their  process  (e.g., 
in  presentations). 

Evaluation  data 

•  Site  information,  appraisal  schedule,  development  organization 
presentation 

This  activity  begins  the  on-site  data  collection  and  consolidation  process.  The 
development  organization's  presentation(s)  may  include  a  process  overview, 
organization  overview,  documentation  overview,  and/or  improvement  plan 
overview. 

Deliver  presentations.  The  entry  briefing  for  the  development  organization’s 
presentation  should  explain  to  the  team: 

•  What  the  organization  does  (without  giving  a  “marketing  pitch”  or  a  recital 
of  their  standard  processes). 

•  The  organizational  structure  (who  does  what),  especially  any  changes  that 
have  occurred  since  the  delivery  of  the  site  information  packet  (Activity  4). 

•  How  responsibility,  accountability,  and  authority  are  managed,  particularly 
in  regard  to  such  items  as  configuration  management,  quality  assurance, 
integration  and  test,  requirements  definition  and  management,  systems 
test,  and  development. 

•  How  the  organization’s  process  integrates  responsibility,  accountability, 
and  authority  through  the  development  life  cycle;  the  organization’s 
description  should  be  focused  on  the  projects  selected  for  review. 

•  The  organization-level  documents  (policies,  procedures,  etc.)  present 
a  roadmap  of  how  the  documents  are  organized.  The  presentation  may 
describe  how  the  organization’s  documentation  is  laid  out  and  how  they 
relate  to  each  other. 

Listen,  ask  questions,  and  tag  notes.  This  activity  is  much  like  an  interview, 
except  the  forum  for  the  participants  to  interact  with  the  team  is  different.  The 
forum  for  the  team  to  collect  data  is  a  formal  presentation  from  the 
organization.  Like  an  interview,  active  listening  is  important.  Clarifying 
questions  are  expected.  The  purpose  of  an  in-brief  is  similar  to  the  initial 
document  review.  The  team  tries  to  get  a  sound  picture  from  a  high  level 
perspective  of  how  the  organization  does  business  from  a  process  perspective. 
For  the  in-brief  to  work  well,  it  is  imperative  that  the  sponsoring  organization 
provide  instructions  to  the  development  organization  as  to  the  expected 
contents  and  format  for  the  in-brief. 
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Outputs 


Outcome 


Options 


Individual  team  member  notes  should  be  “tagged”  as  soon  as  possible,  just  as 
in  an  interview.  Tagging  notes  refers  to  a  shorthand  notation  that  the  team  uses 
to  quickly  identify  data  in  subsequent  data  consolidation  sessions.  Typical 
tagging  includes  noting  how  the  fact  or  strong  inference  written  in  the  notes 
relates  to  the  reference  model  and  whether  it  is  indicative  of  a  strength  or  a 
weakness.  Observations  generated  from  notes  that  come  from  presentations 
like  the  organization  in-brief  can  be  used  to  validate  findings. 

Updated  evaluation  data 

•  site  information,  appraisal  schedule,  terminology,  presentation  slides 
Working  notes 

Requests  for  additional  data 

The  evaluation  team  has  a  refined/updated  understanding  of  the  organization’s 
process  operations. 

The  team  may  receive  a  more  in  depth  “demonstration”  or  “walk  through”  of 
work  processes.  This  presentation  would  be  geared  to  showing  the  team  how 
processes  are  actually  implemented,  using  information  provided  by  the  team, 
in  advance,  on  the  detailed  processes  that  will  be  investigated  on  site.  This 
presentation  would  be  done  on  the  first  morning  of  the  site  visit,  and  could 
mutually  support  the  initial  document  review  activity. 
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3.9  Activity  9  Review  Documents 

Table  3-13  and  Figure  3-10  below  provide  an  overview  of  the  steps  in  this 
activity.  * 


Activity  I  Steps  Outcome 


Review  Step  9A:  Determine  Information  Needed  Understand  processes  actually 

Documents  Step  9B:  Select  or  Request  Documents  implemented  in  the  organization. 

Step  9C:  Review  Documents 
Step  9D:  Take  and  Tag  Notes 

Table  3-13:  Review  Documents 


Figure  3-10:  Review  Documents  Activity  Diagram 
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Purpose 

Inputs 

Action 


Collect  data  by  examining  process  artifacts  (e.g.,  documents). 

Evaluation  data 

•  Site  information,  initial  document  set,  annotated  worksheets/checklists, 
document  review  strategy 

This  activity  is  repeated  multiple  times.  It  is  undertaken  to  understand 
processes  as  actually  implemented  in  the  organization  —  the  track  record  of 
objective  evidence  of  process  use.  The  same  basic  process  is  used  throughout 
the  site  visit,  although  the  type  of  information  sought  will  change  as  the  team 
learns  more  about  the  organization’s  processes  during  the  site  visit.  The  team 
will 


•  Determine  information  needed,  and  if  it  can  be  obtained  from  a  document. 

•  Select  or  request  documents  for  review. 

•  Review  the  documents  for  the  information  needed,  and 

•  Take  notes  and  tag  the  notes  relative  to  the  reference  model  used. 

The  remainder  of  this  section  is  devoted  to  discussing  the  different  types  of 
documents  and  document  review  tasks,  when  documents  are  reviewed,  and 
how  they  relate  to  the  method  goals. 

Three  levels  of  documents  are  reviewed:  «►  organization-level  documents, 
"*•  project-level  documents,  and  «►  implementation-level  documents.  Initial 
document  review  focuses  on  only  the  first  two  levels,  and  can  often  be  started 
prior  to  the  site  visit.  The  “track  record”  documents  are  primarily  reviewed  on 
site.  However,  review  of  organization-  and  project-level  documentation  is  not 
limited  to  the  initial  document  review  period. 

Initial  document  review.  The  documents  for  initial  document  review  were 
requested  during  Activity  7,  Prepare  for  Data  Collection.  These  include  both 
project-  and  organization-level  documents.  The  request  must  be  previously 
coordinated  with  the  development  organization’s  site  technical  coordinator  so 
that  the  documents  will  be  readily  available  in  the  team’s  assigned  work  area. 

The  team  examines  their  strategy  and  planning  documents  to  determine  what 
information  the  process  documents  can  provide.  The  team  then  reviews  the 
initial  document  set.  Initial  document  review  is  focused  on  organization-level 
documents  and  high-level  project  documents. 
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During  the  initial  document  review,  the  team  gains  further  insight  into  each 
scheduled  interviewee’s  proper  role  in  the  organization’s  operations. 
Information  is  collected  as  working  notes.  Initial  document  review  is  usually 
done  before  starting  interviews,  but  may  be  interspersed  with  interviews. 

The  purpose  of  this  step  is  for  the  team  to  determine  the  degree  to  which  the 
organization-level  documents  and  project-level  documents  define  and  support 
standard  processes  for  the  key  process  areas  that  are  under  investigation. 

From  this  activity,  the  team  gains  a  better  understanding  of  the  development 
organization’s  organizational  structure  and  process,  and  is  better  prepared  for 
exploratory  interviews.  By  providing  further  insight  into  the  policies  and 
procedures  that  guide  the  organization’s  processes,  the  team  can  sometimes 
eliminate  the  need  for  a  question  during  the  interviews  or  sharpen  the  focus  for 
a  question. 

Another  objective  of  initial  document  review  is  to  identify  other  documents  from 
the  development  organization  that  may  be  needed  by  the  team;  this  lets  the 
team  seek  clarification  through  the  site  technical  coordinator  or  from  a 
participant  during  interviews. 

The  direct  output  of  document  review  is  a  set  of  working  notes.  The  team  will 
understand  the  purpose  and  content  of  each  relevant  document  and  the 
document’s  relation  to  the  topics  that  the  team  wants  to  evaluate.  Subsequent 
interviews  will  be  better  focused;  the  team  should  have  a  better  idea  of  which 
employees  to  interview  about  each  topic  and  what  to  ask  them. 

Detailed  document  review.  The  team  reviews  project-level  and 
implementation-level  documents  to  validate  information  gathered  through 
other  sources  such  as  interviews  and  higher  level  document  review. 
Documents  on  this  level  provide  an  audit  trail  of  the  processes  used  and  the 
work  performed.  Through  these  reviews,  the  team  confirms  or  negates  the 
proposition  that  the  actual  work  practices  implement  the  processes  described 
in  the  organization-  and  project-level  documents. 

The  purpose  of  this  step  is  to  search  for  objective  evidence  of  how  the 
processes  are  actually  implemented,  and  how  well  they  correlate  to  what  the 
organization  says  is  supposed  to  be  done  —  this  provides  support  for  findings. 
In  other  words,  the  team  determines  whether  the  processes  defined  on  paper 
and  elicited  from  the  interviews  correspond  to  what  the  people  on  the  projects 
are  actually  doing. 
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Evaluation  data  (includes  working  notes) 

Requests  for  additional  data  (e.g.,  documents) 

Understand  processes  actually  implemented  in  the  organization. 

Initial  document  review  can  often  be  conducted  prior  to  the  site  visit.  This  is 
preferable  in  applications  for  internal  evaluation  and  process  monitoring,  and 
may  be  necessary  in  broad  scope  evaluations  in  order  to  make  optimum  use 
of  available  site  visit  time. 

Detailed  document  review  can  be  done  on  an  individual  level,  with  mini-teams, 
and  can  also  be  done  in  parallel  with  consolidation  interviews  (see  Activity  10). 
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3.1 0  Activity  1 0  Conduct  Interviews 

Table  3-14  and  Figure  3-1 1  below  provide  an  overview  of  the  steps  in  this 
activity. 


1  Activity 

1  Steps 

1  Outcome  I 

Conduct 

Interviews 

Step  10A:  Determine  Information  Needed 
Step  10B:  Select  or  Request  Interviewee(s) 
Step  IOC:  Ask  Questions 

Step  1 0D:  Listen 

Step  10E:  Take  and  Tag  Notes 

Understand  site  personnel  perspective  on 
processes  implemented  in  the 
organization. 

Table  3-14:  Conduct  Interviews 


Purpose  Collect  data  by  interviewing  process  agents  (e.g.,  managers,  practitioners,  and 
process  owners). 

inputs  Evaluation  data  which  includes  evaluation  schedule,  site  information,  interview 

strategy,  interview  questions,  working  notes,  annotated  worksheets/checklists. 

Requests  for  documents. 

Action  This  activity  is  repeated  multiple  times.  Interview  types  include  all-on-one,  all- 

on-many,  few-on-one,  and  few-on-few  (where  the  first  term  denotes  number  of 
team  members  and  the  second  term  denotes  number  of  participants.  The  term 
“few”  for  team  members  assumes  a  mini-team  approach,  usually  seen  in 
follow-up  interviews.) 

Determine  information  needed.  The  team  must  always  determine  what 
information  they  need  first.  Determining  what  information  is  needed  is  an 
important  way  to  ensure  that  the  available  (limited)  site  visit  time  allocated  to 
interviews  is  conducted  in  the  most  efficient  way  relative  to  the  evaluation 
goals.  If  there  were  no  subcontractors  on  a  planned  procurement,  for  example, 
the  team  would  not  ask  interviewees  questions  about  the  subcontract 
management  KPA.  Or  if  the  team  reached  consensus  that  they  have  a 
sufficient  number  and  type  of  observations  to  judge  the  “Plan  the  Software 
Project”  goal  (goal  #2)  of  the  Software  Project  Planning  KPA,  they  would  not 
need  to  ask  additional  questions  about  this  area. 

Select  or  request  interviewee(s).  The  team  must  decide  who  might  be  able 
to  provide  needed  information.  The  team  must  work  with  the  site  focal  point  to 
schedule  the  appropriate  managers  and  topic  area  practitioners. 

Some  issues  to  consider  when  deciding  on  interviewing  participants  as 
individuals  (all-on-one  or  few-on-one)  or  as  groups  (all-on-many,  or  few-on- 
few),  include 
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Figure  3-1 1 :  Conduct  Interviews 


•  is  the  individual  a  manager  or  a  practitioner?  (It  is  generally  more 
productive  to  interview  practitioners  in  a  group  setting.) 

•  if  the  person  was  in  the  room  with  others,  would  other  participants  feel 
inhibited  from  an  open  exchange  of  information?  (No  one  should  be  in  a 
group  interview  if  they  will  be  less  likely  to  openly  disclose  information.) 

•  is  the  person  in  the  reporting  chain  of  others  to  be  interviewed?  (To  ensure 
unbiased  responses,  a  manager  should  never  be  in  an  interview  with 
subordinates.) 
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•  is  the  person  an  opinion  leader?  (Opinion  leaders  may  bias  the  input  of 
others  in  a  group.  Often  it  is  more  productive  to  interview  them  separately, 
or  to  take  precautions  in  a  group  setting  to  avoid  letting  the  individual 
dominate  the  session.) 

Ask  questions,  listen,  and  take  and  tag  notes.  The  interviewing  steps  are 
simple,  but  the  activity  clearly  is  not.  Active  listening  and  note  taking  are  two 
critical  skills  that  must  be  mastered  to  maximize  the  amount  of  data  collected 
during  an  interview. 

The  remainder  of  this  section  discusses  interview  sessions,  strategies  for 
interviewing,  and  types  of  interviews  that  are  used  in  the  SCE  method. 

Interview  sessions  include  both  management  and  practitioner  interviews.  The 
baseline  SCE  V3.0  approach  is: 

•  management  interviews  are  all-on-one 

•  group  (practitioner)  interviews  are  all-on-many  (with  no  managers  present) 

•  follow-up  interviews  may  be  few-on-few  or  few-on-one 

Interview  strategies.  One  strategy  is  for  the  team  to  interview  the  organizational 
employees  with  responsibilities  at  the  organization  level,  and  then  the 
employees  with  responsibilities  at  the  project  level.  This  is  a  “top  down” 
strategy.  An  alternative  strategy  is  to  interview  people  by  proceeding  from  one 
project  to  the  next,  and  follow  up  by  interviewing  people  with  organization-level 
responsibilities. 

Another  strategy  is  to  interview  groups  of  individuals  with  similar  functions 
across  projects,  and  across  the  development  life  cycle  (e.g.,  configuration 
management,  design,  test). 

In  any  strategy  used,  the  majority  of  the  time  spent  interviewing  should  be  at 
the  practitioner  level,  since  they  are  closest  to  the  processes  actually  being 
implemented  in  an  organization. 

Interview  types.  SCE  distinguishes  exploratory  interviews  from  consolidation 
interviews.  Exploratory  interviews  have  these  characteristics: 

•  They  provide  insight  into  how  key  process  areas  are  actually  implemented. 

•  They  determine  the  extent  to  which  processes  have  been  internalized. 

•  They  identify  critical  *►  implementation-level  documents. 

•  They  primarily  use  open  ended  and  guided  questioning  techniques3. 
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Interviews  help  the  team  determine  the  extent  to  which  the  documented 
procedures  and  policies  have  been  implemented  throughout  the  projects.  By 
asking  project  personnel  about  specific  practices  (e.g.,  design  and  code 
reviews),  the  team  can  evaluate  whether  the  organization  and  project-level 
policies  and  procedures  have  been  communicated  to  the  people  who  need  to 
implement  them  and  if  they  are  understood. 

Exploratory  interviews  also  point  the  SCE  team  to  the  implementation-level 
documentation  for  a  project  and  guide  the  document  review  at  that  level.  This 
documentation  is  used  to  validate  both  the  interview  responses  and  the  higher 
level  procedures  during  subsequent  document  reviews. 

Consolidation  interviews  are  specific  sessions  with  people  who  have 
previously  been  interviewed,  or  who  were  identified  as  having  specific 
information  needed  by  the  team.  Consolidation  interviews  differ  from 
exploratory  interviews  in  the  following  ways: 

•  they  are  principally  focused  on  validating  data  already  gathered 

•  they  are  focused  on  seeking  specific  information  needs 

•  they  use  more  directed  questioning  techniques  to  obtain  the  information 

•  they  may  be  conducted  by  a  subset  of  the  entire  team 

•  they  can  run  in  parallel  with  other  consolidation  interviews 

Every  piece  of  information  obtained  during  an  interview  can  lead  to 
identification  of  a  strength,  a  weakness,  or  an  improvement  activity.  Before 
information  can  be  transformed  into  a  finding,  it  must  be  validated.  Central  to 
the  validation  process  is  the  accurate  transformation  of  the  raw  data  obtained 
in  an  interview.  Active  listening  is  required  to  transcribe  data  directly  in  the  form 
that  it  was  provided,  without  inserting  personal  judgments  or  biases.  The  first 
step  in  the  transformation  is  to  “tag”  individual  notes,  citing  relevant  items  such 
as  mapping  to  the  reference  model,  and  noting  potential  strengths  and 
weaknesses.  Tagging  notes  is  the  link  between  the  data  collection  activity  and 
the  consolidation  activity.  Taking  sound  notes  that  are  tagged  well  will  assist 
the  transformation  of  notes  into  observations  during  consolidation  (see  Activity 
11). 

Outputs  Evaluation  data  (includes  working  notes) 

Requests  for  additional  data  (e.g.,  documents,  interviewees) 


using  guided  questioning  techniques  focuses  the  interviewees  on  the  topic  areas  important  to  the  team  rather 
than  simply  letting  them  discuss  less  relevant  areas. 
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Outcome 


Options 


Understand  site  personnel  perspective  on  processes  implemented  in  the 
organization. 

In  SCEs  for  Acquisition,  procuring  officials  may  decide  that  interviews  must  be 
all-on-one  rather  than  all-on-many  due  to  regulatory  interpretations.  All-on-one 
interviews  are  not  the  recommended  style  for  collecting  data  from  practitioners. 
Group  interviews  of  functional  and  staff  engineers  allow  the  team  to  collect  a 
greater  quantity  of  data  in  a  much  shorter  period  of  time.  They  can  also 
increase  the  quality  of  the  data  by  eliminating  unrepresentative  data  coming 
from  an  individual  who  may  be  new  to  the  organization,  overly  nervous  about 
the  interview,  etc. 

The  team  may  break  into  parts,  “mini-teams,”  to  conduct  parallel  follow  up  or 
consolidation  interviews.  The  important  principle  is  to  adhere  to  the  rule  of 
“multiple  eyes  and  ears.”  A  team  always  wants  more  than  one  individual 
participating  in  an  interview.  It  helps  ease  the  burden  on  the  interviewer,  and 
also  helps  mitigate  potential  recall  and  analysis  errors  during  consolidation. 
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3.11  Activity  11  Consolidate  Data 

Table  3-15  and  Figure  3-12  below  provide  an  overview  of  the  steps  in  this 
activity. 


1  Activity 

1  Steps 

1  Outcome  1 

Consolidate 

Step  11  A:  Organize  and  Combine  Data 

The  team  has  an  agreed  to  baseline  of 

Data 

-  Assign  KPA  Monitor 

information  known,  information  needed, 

-  Record  observation 

and  the  strategy  to  obtain  needed 

-  Consolidate  observations 

Step  1 1 B:  Determine  Data  Sufficiency 

-  Perform  coverage  check 

-  Validate  observations 

-  Judge  sufficiency  of  data  for  rating 

-  Review  consolidation  work 

Step  11C:  Review  and  Revise  Data 
Gathering  Plan 

-  Identify  information  needed 

-  Consolidate  information  needed 

-  Revise  data  gathering  plans 

information. 

Table  3-15:  Consolidate  Data 


Purpose  Transform  the  data  collected  into  formal  team  observations  of  process 

strengths  and  weaknesses  relative  to  the  reference  model  (e.g.,  the  CMM). 

inputs  Evaluation  data  (ex.  working  notes,  annotated  worksheets/checklists, 

annotated  draft  findings  presentation) 

Action  This  activity  is  repeated  multiple  times.  Consolidation  of  data  is  the  crux  of  the 

data  collection  activity.  In  this  activity  collected  data  is  analyzed  and  prepared 
for  rating. 

In  the  Activity  1 1  sub-steps  listed  in  the  table,  the  first  three  are  individual  key 
process  area  monitor  or  mini-team  activities;  the  last  seven  are  mini-team  or 
whole  team  activities. 

A  critical  requirement  that  must  be  met  by  this  activity  is  to  make  data  visible  to 
the  entire  team  for  consensus.  An  additional  SCE  design  constraint  is  to 
provide  artifacts  (forms)  that  can  be  populated  with  information  in  both 
automated  and  non-automated  fashion. 
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Figure  3-12:  Consolidate  Data  Activity  Diagram 

Organize  and  combine  data.  Working  notes  that  are  used  for  consolidation 
come  from  multiple  sources: 

•  Instrument  analysis  (Activity  5) 

•  Organizational  presentations  (Activity  8) 

•  Document  review  (Activity  9) 

•  Interviews  (Activity  10) 

•  Draft  findings  presentation  (Activity  12) 
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'*»  Consolidation  is  the  decision  making  activity  in  the  iterative  information 
gathering  and  decision  making  process.  Consolidation  has  three  primary 
objectives: 

•  Organizing  the  information  obtained  from  data  gathering  sessions  and 
combining  it  into  a  manageable  summary  of  data. 

•  Determining  whether  or  not  the  information  provides  a  sufficient  basis  for 
making  judgments  concerning  an  organization’s  process  capability. 

•  If  not,  determining  any  revisions  that  should  be  made  to  the  data  collection 
plan  to  obtain  the  additional  information  required  to  make  judgments. 

Quality  control  checks  are  built  into  the  consolidation  process  as  part  of  the 
judgments  made  relative  to  the  first  two  objectives. 

During  consolidation,  the  team  assesses  their  progress  toward  the  goal  of 
validating  topics.  No  particular  format  is  specified  for  the  caucus,  but  the 
following  steps  are  typical: 

•  The  team  members  review  the  topics  focused  on  and  the  notes  that 
resulted  from  the  most  recent  data  collection  activities.  Observations  are 
generated  and/or  combined  with  others  by  team  members  and/or  mini¬ 
teams. 

•  The  team  reviews  any  new  observations,  and  identifies  areas  that  require 
further  clarification.  The  team  validates  the  observations,  meaning  they 
agree  that  an  observation  is  accurate,  and  that  they  have  reached 
consensus  on  it  (accepted,  rejected,  or  modified  the  observation). 

•  Subsequent  team  consensus  determines  the  sufficiency  of  the  data  for 
rating  (observations  that  are  validated,  cover  the  area,  and  are 
corroborated).  Sufficient  observations  automatically  become  candidate 
findings  for  delivery  in  the  draft  findings.  (See  Activity  12.) 

•  If  the  team  cannot  reach  consensus  at  any  point  in  the  consolidation 
process,  they  identify  what  information  is  needed  to  resolve  the  outstanding 
issues,  and  generate  a  plan  for  collecting  the  information. 

•  If  enough  information  has  been  gathered  to  make  a  determination  about  a 
topic,  it  is  dropped  from  further  consideration  unless  subsequent  data 
collection  related  to  other  topics  reveals  new  information  on  the  “closed” 
topic.  For  example,  if  no  subcontractors  are  used  on  the  projects,  findings 
related  to  subcontractor  management  would  not  be  applicable,  leading  to  a 
determination  of  “not  applicable.”  (See  Activity  13.) 


Purpose.  The  purpose  of  the  team  consolidation  is  to  analyze,  share,  and 
consolidate  the  available  information  in  order  to  reach  conclusions  about  the 
topics.  The  SCE  team  gathers  a  large  quantity  of  data;  caucusing  helps  the 
team  sift  through  the  information.  Consolidating  also  provides  a  chance  for  the 
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team  to  share  diverse  perspectives  on  the  data,  which  helps  prevent 
misinterpretations  and  premature  decisions.  Consolidation  keeps  the  team 
focused  on  the  evaluation  objectives. 

The  standard  consolidation  tasks  that  occur  throughout  the  site  visit  are: 

•  Team  members  or  mini-teams  determine  potential  strengths,  weaknesses, 
or  improvement  activities  based  on  the  available  information  and  annotate 
the  information  as  '**■  observations  on  key  process  area  worksheets, 
(organize  and  combine  data) 

•  Mini-teams  or  the  whole  team  validate  observations  based  on  what  was 
heard  or  seen  since  the  last  consolidation  session. 

•  The  team  determine  data  sufficiency  for  rating  purposes. 

•  The  team  identifies  a  need  for  more  data  to  confirm  or  negate  an 
observation  about  one  or  more  topics.  This  results  in  updated  data 
collection  plans  —  generating  requests  for  additional  documentation  to 
review  or  additional  interviews.  (Review  and  revise  data  gathering  plans.) 

Judgments  made  during  data  transformations.  There  are  many  judgments  that 
team  members,  mini-teams,  and  the  whole  team  make  throughout  the  site  visit. 
In  order  to  be  validated,  observations  must  be  determined  to  be  accurate. 
Accuracy  of  observations  is  continually  checked  by  the  team  by  evaluating 

•  wording 

•  basis  (i.e.,  facts  and  strong  inferences) 

•  categorization  (against  reference  model  components  or  non-reference 
model  areas) 

•  classification  (as  strengths,  weaknesses,  or  improvement  activities) 

•  relevance  and  significance  (only  the  most  important  and  non-redundant 
items  should  be  transformed  into  findings  —  the  consolidation  process 
should  “weed  out”  unimportant  data  relative  to  meeting  evaluation  goals.) 

The  judgments  required  to  derive  observations  from  notes  include: 

•  Determining  that  each  observation  is  worded  appropriately  (i.e.,  is  clear, 
concise,  does  not  make  use  of  absolutes,  is  expressed  in  site  terms,  and 
maintains  confidentiality  principles). 

•  Determining  that  each  observation  is  based  on  facts  documented  in  notes 
or  on  strong  inferences  drawn  from  those  facts. 

•  Determining  that  each  observation  is  relevant  by  determining  that  it  can  be 
categorized  in  terms  of  the  reference  model  or  otherwise  has  a  significant 
impact  on  the  organization’s  process  capability. 
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•  Determining  that  each  observation  is  significant  by  determining  that  it  can 
be  classified  as  evidence  of  strength,  weakness,  acceptable  alternative 
processes,  or  non-applicable  processes. 

•  Determining  that  the  set  of  observations  does  not  contain  redundancies. 


Determine  data  sufficiency.  Successful  consolidation  depends  on  the  team’s 
consensus-building  ability.  Data  sufficiency  is  a  result  of  the  team  reaching 
consensus  on  observations  regarding  their  validity,  coverage,  and 
corroboration. 

Judgments  required  to  validate  observations  as  findings  include: 

•  Determining  that  each  observation  is  valid  by  reaching  consensus  on  the 
accuracy  of  the  item  (i.e.,  wording,  factual  basis,  relevance,  significance, 
and  non-redundancy). 

•  Determining  whether  or  not  an  observation  is  consistent  with  other  findings 
(i.e.,  validated  observations). 

•  Determining  whether  the  set  of  observations  fully  cover  the  area  of 
investigation  (to  the  extent  required  by  the  rating  baseline  option). 

•  Determining  whether  or  not  a  particular  observation  is  sufficiently 
corroborated  (i.e.,  having  enough  of  the  right  kind  and  sources  of  data  for 
observations). 

Rules  of  corroboration.  In  order  for  an  SCE  finding  (strength,  weakness,  or 
improvement  activity)  to  exist,  the  following  data  sufficiency  guidelines  must  be 
met: 

•  There  must  be  objective  evidence  in  the  form  of  documentation  to  support 
the  finding  (lack  of  documentation,  if  sought  out  by  the  team,  constitutes 
objective  evidence). 

•  The  team  must  observe  supporting  evidence  from  two  or  more  sources,  in 
independent  sessions. 

•  The  supporting  evidence  must  include  observations  generated  from 
interviews  or  presentations  with  the  people  who  perform  the  related 
process,  and  reviews  of  documentation  that  result  from  executing  that 
process 

•  The  team  must  generate  the  findings  through  a  consensus  process.  That 
is,  there  are  no  minority  opinions  opposed  to  the  finding. 

•  The  evidence  must  support  the  findings. 

All  judgments  made  by  the  team  should  be  confirmed  by  at  least  two  separate 
pieces  of  information.  As  the  significance  of  the  judgment  increases,  the  team 
may  decide  that  confirmation  from  three  or  more  separate  sources  of 
information  are  needed.  An  example  might  be  that,  during  consolidation,  the 
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team  realizes  that  a  particular  observation  is  significant  enough  that  it  may 
cause  a  key  process  area  to  be  rated  unsatisfied  (see  Activity  13,  Make  Rating 
Judgments).  In  that  case,  they  may  decide  that  an  observation  which  had  an 
instrument  as  a  source  of  the  data  is  not  sufficient  to  validate  it.  As  a  general 
rule,  if  there  is  any  doubt  at  all  about  whether  a  finding  is  valid,  the  team  should 
defer  it  to  the  consolidation  step  and  should  initiate  additional  data  collection 
efforts. 

Coverage  rules.  Determining  data  sufficiency  is  the  consolidation  step  where 
previous  decisions  regarding  coverage  requirements  are  critical  (see 
determining  the  rating  baseline  in  Activity  1).  The  following  guidance  is 
provided  to  ensure  adequate  coverage  of  model  components  relative  to  the 
rating  baseline  chosen  —  preparing  the  team  for  the  rating  process  to  come 
(see  Activity  13): 

•  A  maturity  level  is  covered  if  all  of  its  key  process  areas  and  those  of  all 
lower  levels  are  covered. 

•  A  key  process  area  is  covered  if  all  of  its  goals  are  covered. 

•  A  goal  is  covered  if  all  of  its  subordinate  model  components  are  covered 
(key  practices  of  the  activities  performed  and  institutionalized  common 
features) 

•  A  common  feature  is  covered  if  sufficient  observations  exist  to  judge  the 
extent  of  institutionalization  of  the  common  feature. 

•  A  key  practice  is  covered  when  the  team  judges  that  the  key  aspects  of  the 
item  are  covered  by  the  set  of  observations  related  to  it. 

When  rating  decisions  are  made  based  on  data  that  does  not  completely  cover 
the  area  of  investigation,  risk  in  the  appraisal  output  increases  and  the  ratings 
must  be  accompanied  by  an  associated  coverage  factor  which  documents  the 
CMM  components  that  have  been  covered.  This  will  allow  sponsors  to  make 
well  informed  decisions  using  the  available  appraisal  outputs.  All  coverage 
decisions  must  consider  the  nature  of  the  findings  relative  to  the  reference 
model,  the  appraised  entity,  and  the  appraised  entity’s  lifecycle(s).  Explicit 
examination,  documentation,  and  acceptance  of  alternative  practices  to  the 
reference  model  must  enter  into  this  consolidation  process. 

Review  and  revise  data  gathering  plan.  If  an  observation  cannot  be 
validated,  if  doubt  remains,  or  if  consensus  is  not  achieved  despite  additional 
documentation  or  interviews,  then  there  can  be  no  finding.  The  team  must 
determine  what  information  is  needed  in  order  to  resolve  the  disagreement, 
and  revise  their  plans  to  obtain  the  needed  information. 


CMU/SEI-96-TR-002 


©  1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


95 


Activity  Descriptions 


April  1996 


Outputs 


Outcome 


Options 


If  the  team  identifies  an  observation  that  is  a  weakness,  the  development 
organization  (through  the  site  visit  coordinator  or  in  subsequent  interviews) 
should  be  given  an  opportunity  to  produce  evidence  that  might  mitigate  or 
refute  the  response  that  indicated  a  weakness.  (See  also  Activity  12,  Deliver 
Draft  Findings.)  By  double  checking,  the  team  avoids  making  findings  based  on 
anomalous  responses.  The  request  for  clarification  should  define  the  subject 
matter  and  ask  if  what  the  team  observed  or  heard  is  representative.  For 
example,  the  team  might  ask  “We  were  not  able  to  determine  if  the  estimates 
for  project  size  were  based  on  actual  data.  Did  we  miss  something?” 

Summary.  It  is  possible  for  a  given  component  of  the  reference  model  to  have 
strengths,  weaknesses  and  improvement  activities  exhibited  at  the  same  time 
—  for  example,  well-defined  procedures  (a  strength),  no  training  in  the 
procedures  (a  weakness),  and  an  ongoing  course  development  effort  for  the 
new  procedures  sponsored  by  the  organization  (an  improvement  activity). 

Evaluation  data 

•  Observations  (reference  model,  non-reference  model) 

•  revised  data  collection  plan  (document  review  and  interview  strategies) 

•  annotated  worksheets/checklists, 

Requests  for  additional  data  (additional/  new  interviewees  or  documents) 

The  team  has  an  agreed  to  baseline  of  information  known,  information  needed, 
and  the  strategy  to  obtain  needed  information. 

Consolidation  can  be  done  with  individual  key  process  area  monitors,  or  can 
be  done  with  KPA  mini-teams.  In  applications  using  small  evaluation  teams, 
individual  monitors  are  likely.  In  large  scope  applications,  with  large  teams,  the 
mini-team  approach  is  most  prevalent.  Many  people  find  the  mini-team 
approach  to  be  the  best  way  to  provide  a  “buddy”  system  of  data  collection, 
analysis,  and  corroboration.  Having  multiple  people  responsible  for  KPAs 
ensures  that  important  information  is  not  missed  by  the  team,  either  during  data 
collection  activities,  or  during  the  consolidation  activity. 

The  baseline  SCE  process  supports  using  KPA  worksheets  for  annotating, 
tracking,  and  consolidating  observations.  This  technique  is  most  easily 
automated.  However,  many  teams  are  comfortable  using  KPA  “wall  charts”  to 
perform  the  same  consolidation  functions.  This  technique  is  beneficial  in  a 
paper  intensive  process,  and  meets  requirements  for  a  good  observation 
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tracking  system.  Both  techniques  can  work  well.  An  important  consideration  in 
this  decision  is  how  the  consolidated  information  is  made  visible  to  the  whole 
team  for  tasks  that  require  all  team  members  to  participate. 
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3.12  Activity  12  Deliver  Draft  Findings 

Table  3-16  and  Figure  3-13  below  provide  an  overview  of  the  steps  in  this 
activity. 


I  Activity 

1  Steps 

1  Outcome 

Deliver  Draft 
Findings 

Step  12A:  Prepare  Draft  Findings 

Presentation 

Step  12B:  Present  Draft  Findings  And  Solicit 
Feedback 

Step  12C:  Listen 

Step  12D:  Take  and  Tag  Notes 

The  quality  of  the  evaluation  data  and 
results  is  improved,  and  credibility  and  buy- 
in  to  the  appraisal  process  and  its  results  is 
generated. 

Table  3-16:  Deliver  Draft  Findings 

Purpose 

Validate  observations  and  collect  data  by  conducting  interactive  feedback 
sessions  with  participants. 

Inputs 

Evaluation  data 

•  annotated  worksheets/checklists 

•  observations 


Action  This  activity  includes  preparation  and  presentation  of  the  draft  findings. 

Information  obtained  during  the  draft  findings  presentation  will  be  used  as 
inputs  during  development  of  the  final  findings. 

In  the  Deliver  Draft  Findings  activity,  the  team  must  prepare  a  briefing,  deliver 
the  briefing,  and  actively  solicit  feedback  from  the  participants.  Team  members 
must  take  and  tag  notes  like  other  data  collection  activities.  An  important 
aspect  of  the  draft  findings  session  is  that  it  is  both  a  data  validation  and  data 
collection  activity.  Presenting  observations  to  the  participants  is  a  data 
validation  task.  Participant  feedback  on  the  validity  of  the  observations  is 
sought.  Asking  questions  of  participants  when  they  react  to  the  briefing,  or 
when  seeking  information  still  needed  to  validate  an  observation,  is  a  data 
collection  task. 

Prepare  draft  findings  presentation.  The  baseline  SCE  Method 
recommends  delivering  draft  findings  of  weaknesses  only.  This  assumes  that 
participants  will  not  quarrel  strongly  over  perceived  and  validated  strengths. 
This  baseline  approach  optimizes  time  overall  and  ensures  maximum  time  is 
spent  validating  weaknesses  and  collecting  additional  data  to  validate  these 
and  potentially  new  items. 
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Figure  3-13:  Deliver  Draft  Findings  Activity  Diagram 


If  the  Consolidate  Data  activity  is  done  properly,  preparing  the  briefing  is 
simple.  Observations  that  the  team  has  reached  consensus  on  regarding 
validity  and  sufficiency  are  candidates  for  briefing  in  the  draft  findings 
presentation.  If  worded  well,  the  observation  can  be  transcribed  directly  onto  a 
briefing  chart  because  a  validated  observation,  by  definition,  is  a  finding.  (See 
the  Glossary.) 

Present  draft  findings  and  solicit  feedback.  The  presenter  is  chartered  with 
the  task  of  actively  soliciting  feedback  from  the  participants.  The  team  leader 
may  present  the  draft  findings,  or  KPA  monitors  may  be  tasked  to  brief  their 
respective  areas.  Setting  expectations  at  the  beginning  of  the  session  is 
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critical.  Participants  need  to  understand  the  purpose  of  the  session,  and  that 
only  weaknesses  are  shown.  They  need  to  know  that  a  significant  amount  of 
additional  data  will  be  fed  back  to  them  in  the  final  findings  presentation 
(Activity  14).  There  are  always  multiple  feedback  sessions  at  this  point  in  the 
process.  Practitioner  participants  and  management  participants  are  always 
briefed  separately  at  this  stage,  to  continue  facilitating  the  most  open  exchange 
of  information  between  the  participants  and  the  team. 

Listen  and  take  notes.  Active  listening  is  important  to  this  activity,  in  the  same 
manner  as  in  Activity  10,  Conduct  Interviews.  The  team  is  looking  for  positive 
or  negative  affirmation  of  the  findings  presented.  The  team  must  solicit 
feedback.  The  session  doesn’t  work  if  the  participants  don’t  respond.  It  is 
important  to  note  non-verbal  as  well  as  verbal  feedback.  Working  notes  from 
feedback  sessions  include  annotations  made  directly  on  the  draft  findings 
slides  during  the  presentation  delivery,  in  addition  to  individual  team  member 
notes.  Team  members  tag  their  individual  notes  as  in  all  other  data  collection 
activities. 

Evaluation  data 

•  working  notes 

•  draft  findings  presentation 

The  quality  of  the  evaluation  data  and  results  is  improved,  and  credibility  and 
buy-in  to  the  appraisal  process  and  its  results  is  generated. 

A  team  may  choose  to  use  one  of  the  following  briefing  formats: 

•  one  person  brief  all  of  the  draft  findings  for  all  sessions, 

•  one  person  brief  all  of  the  draft  findings,  but  use  a  different  person  for  each 
session,  or 

•  multiple  team  members  brief  sections  of  the  draft  findings,  based  on  their 
team  roles/expertise  areas,  during  each  session. 

A  team  may  choose  to  brief  strengths  during  this  presentation  also,  but  there 
are  trade  offs  with  available  site  visit  time.  The  team  leader  should  make  this 
decision  as  part  of  planning  to  meet  the  intent  of  the  evaluation  goals.  Or  the 
team  could  choose  to  decide  to  use  this  option  during  the  evaluation  if  data 
collected  suggests  that  it  would  be  better  to  also  brief  strengths  (again  to 
ensure  appraisal  goals  are  met). 
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This  activity  may  not  be  allowed  in  acquisition  applications  due  to  contractual 
and  legal  constraints  in  the  procurement  process.  (See  SCE  V3.0 
Implementation  Guide  for  Supplier  Selection  for  Acquisition.) 
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3.13  Activity  13  Make  Rating  Judgments 

Table  3-17  and  Figure  3-14  below  provide  an  overview  of  the  steps  in  this 
activity. 


Activity 

Steps 

Outcome 

Make  Rating 
Judgments 

Step  13A:  Judge  Satisfaction  of  Key 
Practices  (if  selected  in  planning). 

Step  13B:  Judge  Satisfaction  of  Common 
Features  (if  selected  in  planning). 

Step  13C:  Judge  Satisfaction  of  the  KPA 
Goals  Based  on  Implementation  and 
Institutionalization. 

Step  1 3D:  Judge  Satisfaction  of  KPAs 

Step  13E:  Determine  Maturity  Level 

A  formal  rating  decision  for  each  reference 
model  component  which  was  planned  to 
be  rated,  and  for  which  the  team  obtained 
sufficient  data  to  meet  method  rules  for 
conducting  the  rating. 

Table  3-17:  Make  Rating  Judgments 


Purpose  Make  decisions  about  the  organization’s  process  capability,  based  on 
validated  observations,  relative  to  the  reference  model  components 
investigated. 

inputs  Evaluation  data 

•  annotated  worksheets/checklists 

•  working  notes 

Action  This  activity  is  the  team  task  of  deciding  on  the  appraised  entity’s  satisfaction 

of  reference  model  components,  based  on  validated  observations  (findings) 
collected  by  the  team.  Decisions  on  what  model  components  to  rate  were 
made  during  requirements  analysis  and  planning  in  Activities  1  and  2.  This 
critical  decision  was  made  between  the  sponsor  and  the  team  leader.  A 
sponsor  could  decide  not  to  have  any  ratings  provided  as  appraisal  outputs. 

Ratings  are  based  on  the  components  defined  in  the  reference  model.  The 
rating  process  proceeds  in  a  bottom  up  manner.  If  selected  in  planning,  key 
practices  and  common  features  will  be  rated  first.  Goals  are  then  rated,  and  the 
results  are  rolled  up  into  KPA  and  maturity  level  ratings.  Rating  decisions  are 
always  made  by  the  team  as  a  whole.  Recommendations  may  be  made  by 
mini-teams,  but  a  consensus  process  must  be  invoked  for  all  final  rating 
judgments. 

In  SCE  V3.0,  using  the  CMM  VI. 1  as  the  reference  model,  the  baseline 
process  includes: 
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Figure  3-14:  Make  Rating  Judgments  Activity  Diagram 


Judge  satisfaction  of  the  goals.  To  fully  satisfy  a  goal,  an  organization  must 
both  implement  a  process  and  also  institutionalize  its  use.  Thus  both  the 
activities  (implementation)  and  other  common  features  (institutionalization) 
must  be  satisfied  for  the  goal  to  be  satisfied.  The  team  must  judge  whether 
each  goal  has  been  both  implemented  and  institutionalized. 

For  example,  resources  are  related  to  the  ability  to  perform  common  feature. 
Resources  are  not  typically  assigned  to  a  KPA  based  on  an  individual  goal. 
Rather,  resources  are  assigned  to  work  efforts  based  on  the  need  to  meet  all 
of  the  goals  of  that  area.  The  notion  of  institutionalization  aspects  cutting 
across  the  entire  set  of  KPA  goals  was  also  reflected  in  how  the  team  initially 
prepared  to  collect  data  on  the  appraised  entity,  using  the  KPA  matrix.  (See 
Activity  7.) 
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A  goal  is  satisfied  if  the  associated  findings  indicate  that  this  goal  is 
implemented  and  institutionalized  either  as  defined  in  the  reference  model,  with 
no  significant  weaknesses,  or  that  an  adequate  alternative  exists. 

Judge  satisfaction  of  the  KPAs.  This  is  a  simple  decision-making  task.  All  of 
the  goals  for  a  specific  KPA  must  be  satisfied  in  order  for  the  KPA  to  be 
satisfied. 

Determine  maturity  level.  This  is  a  simple  decision  making  task.  All  of  the 
KPAs  within  a  maturity  level  and  within  each  lower  level  must  be  satisfied  in 
order  to  be  rated  at  that  maturity  level.  Rating  a  maturity  level  is  an  optional 
output.  Recall  that  maturity  level  can  only  be  rated  when  the  subordinate 
components  making  up  that  maturity  level  have  been  rated  in  accordance  with 
the  rules  regarding  the  rating  baseline  option  selected  (see  Activity  1).  The 
decision  to  rate  maturity  level  must  be  made  during  requirements  analysis  and 
planning,  along  with  determining  the  rating  baseline  option. 

Optional  steps  include  judging  satisfaction  of  key  practices  and  common 
features. 

The  remainder  of  this  section  discusses  what  rating  is,  when  it  can  be 
performed,  how  appraisal  risk  is  measured,  and  what  values  it  takes  on. 

"«*•  Rating  is  performed  for  any  component  for  which  «*■  coverage  and 
"•^validation  and  »«►  corroboration  rules  have  been  met  (see  Activity  11). 
Rating  a  component  is  only  performed  when  method  rules  have  been  met. 

Coverage  generally  means  the  extent  of  “completeness”  in  the  investigation  of 
an  item,  and  can  be  considered  a  “bottom-up”  exercise.  For  example,  if  a 
practice  called  for  the  use  of  a  documented  procedure,  incomplete  coverage  of 
a  practice  might  be  that  a  team  found  a  “procedure”  being  used,  but  was  not 
able  to  investigate  the  “documented”  part  of  this  practice. 

Investigating  a  sampling  of  practices  from  a  KPA  constitutes  incomplete 
coverage  of  that  KPA  and  requires  that  a  coverage  factor  be  reported. 
Similarly,  a  sampling  of  practices  from  within  a  common  feature,  such  as  the 
ability  to  perform  (in  the  CMM  VI. 1),  constitutes  incomplete  coverage.  A 
coverage  factor  should  be  provided  with  the  common  feature  rating,  also. 

The  coverage  factor  measure  is  the  absolute  number  and  percentage  of 
practices  fully  covered  within  the  rating  component.  In  addition,  reports  must 
document  the  actual  CMM  components  that  have  been  covered.  For  example, 
in  the  Software  Project  Planning  KPA  there  are  15  key  practices  of  the  activities 
performed  common  feature.  If  a  team  planned  for  and  completed  full  coverage 


104 


©  1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


CMU/SEI-96-TR-002 


April  1996 


Activity  Descriptions 


of  12  of  the  15  activities  performed  key  practices,  that  number  (12)  and  the 
associated  percentage  coverage  (80%)  as  well  as  the  specific  activities  that 
have  been  covered  would  be  placed  directly  on  the  form  or  chart  showing  the 
ratings.  This  concept  applies  to  the  practice  of  the  other  common  features, 
also. 

All  of  the  goals  of  a  KPA  must  be  satisfied  in  order  for  the  KPA  to  be  satisfied. 
If  a  team  chose  during  planning  to  not  investigate  all  goals  in  a  KPA,  then  the 
KPA  could  not  be  formally  rated.  Similarly,  each  KPA  within  a  maturity  level  and 
each  lower  maturity  level  must  be  rated  in  order  to  determine  whether  the 
maturity  level  has  been  attained.  A  mapping  of  the  CMM  VI  .1  key  practices  to 
the  KPA  goals  is  provided  in  the  SCE  training  materials  as  a  work  aid  for  this 
process. 

Rating  values  of  reference  model  components  are 

•  satisfied 

•  not  satisfied 

•  not  applicable 

•  not  rated 

A  reference  model  component  is  satisfied  if  it  is  implemented  and 
institutionalized  either  as  defined  in  the  reference  model,  or  with  an  adequate 
alternative.  The  rating  scale  for  maturity  levels  in  the  CMM  for  Software,  VI. 1 
is  an  integer  1-5. 

A  reference  model  component  is  unsatisfied  if  there  are  significant  weaknesses 
in  the  appraised  entity’s  implementation  or  institutionalization  of  the  item,  as 
defined  in  the  reference  model,  and  no  adequate  alternative  is  in  place. 

A  reference  model  component  is  “not  applicable”  if  it  does  not  apply  in  the 
organization’s  environment.  A  reference  model  component  is  “not  rated”  if  the 
evaluation  data  does  not  meet  coverage  requirements  or  the  item  is  outside  the 
appraisal  scope. 

Rating  hierarchy.  Figure  4-20  depicts  the  rating  output  hierarchy  used  in  the 
method.  Recall  that  the  findings  of  strengths,  weaknesses,  and  improvement 
activities  for  each  KPA  are  generated  during  the  final  consolidation  activity  (see 
Activity  11).  The  findings  are  the  data  element  used  to  determine  the 
satisfaction  of  reference  model  components. 
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The  SCE  V3.0  rating  output  hierarchy  is  designed  to  support  evolution  to 
models  with  the  same  architecture  as  the  CMM  for  Software  VI. 1.  (e.g., 
TCMM). 

Figure  3-4  should  be  read  bottom-up.  The  lowest  level  components  that  were 
chosen  in  planning  are  rated  first,  and  the  results  roll  up  into  higher  level 
component  judgments.  Recall  that  the  components  rated  depend  on  the 
reference  model  scope  and  rating  baseline  decisions  made  in  Activity  1 , 
Analyze  Requirements. 


Figure  3-15:  Overview  of  the  SCE  V3.0  Rating  Output  Hierarchy 


Typical  outputs  are  strengths,  weaknesses,  and  improvement  activities  against 
each  KPA  investigated.  KPA  profiles  will  also  be  generated  if  ratings  have  been 
completed.  A  KPA  profile  is  a  graphical  representation  of  the  organization’s 
process  capability  in  terms  of  the  model  components  that  were  rated 

Evaluation  outputs  which  include  ratings  of  reference  model  (e.g.,  CMM) 
components. 

Evaluation  data  which  includes  annotated  worksheets/checklists. 

A  formal  rating  decision  for  each  reference  model  component  which  was 
planned  to  be  rated,  and  for  which  the  team  obtained  sufficient  data  to  meet 
method  rules  for  conducting  the  rating. 


106  ©  1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


CMU/SEI-96-TR-002 


April  1996 


Activity  Descriptions 


Options 


It  is  always  an  option  of  the  sponsor  to  forego  any  ratings  at  all,  and  to  simply 
have  the  outputs  be  the  findings. 

The  method  offers  two  options  for  rating  (see  Activity  1 ,  determining  the  rating 
baseline): 

1 .  Depth-oriented  —  partial  scope,  complete  coverage. 

When  an  evaluation  calls  for  a  reduced  scope  investigation  —  in  the  model 
and/or  in  the  organizational  scope  (similar  to  SCE  V2.0)  —  the  default 
approach  is  to  fully  cover  and  rate  those  model  components  that  are  specified 
during  requirements  analysis  and  planning  [Activities  1  and  2], and  that  meet 
method  rules  for  rating.  Performing  a  maturity  level  rating  cannot  be  done  using 
this  option,  and  complete  coverage  of  specified  items  is  required. 

2.  Breadth-oriented  —  full  scope. 

When  a  “full”  model  and  organizational  scope  investigation  is  called  for  (e.g., 
all  model  components  within  several  maturity  levels  and  many  projects 
covering  a  wide  diversity  of  organizational  business).  There  are  two  sub¬ 
options  available.  Either 

•  obtain  complete  coverage  of  model  components  prior  to  rating,  or 

•  report  coverage  factors,  to  be  delivered  with  the  rating  when  complete 
coverage  is  not  obtained. 

Performing  a  maturity  level  rating  may  be  done  using  this  option,  and  complete 
coverage  may  or  may  not  be  required.  However,  the  coverage  factor  should 
always  be  reported  when  rating  decisions  have  been  made  without  complete 
coverage.  A  maturity  level,  may  only  be  determined  when  all  KPAs  within  a 
level  and  all  lower  levels  have  been  rated.  A  KPA  may  be  rated  when  a 
sampling  of  practices  from  each  of  the  goals  of  that  KPA  are  covered. 
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3.14  Activity  14  Deliver  Final  Findings 

Table  3-18  and  Figure  3-16  below  provide  an  overview  of  the  steps  in  this 
activity. 


Activity  Steps 


Outcome 


Deliver  Final  Step  14A:  Prepare  Final  Findings 

Findings  Presentation 

Step  14B:  Present  Final  Findings 
Step  14C:  Close  Out  Site  Activities 


The  sponsor  and  the  appraised 
organization  understand  and  accept  the 
team’s  findings. 


Table  3-18:  Deliver  Final  Findings 


Figure  3-16:  Deliver  Final  Findings  Activity  Diagram 
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Purpose 


Inputs 


Action 


Provide  a  clear  and  actionable  summation  of  the  evaluation  results  to  the 
sponsor  and  the  organization. 

Evaluation  data 

•  annotated  worksheets/checklists 

•  ratings 

This  activity  includes  preparation  and  presentation  of  the  final  findings.  It  also 
includes  site  close  out  activities  such  as  assignment  of  report  writing  and 
follow-on  activities.  If  a  reference  model  component  is  reported  as  unsatisfied, 
there  must  be  corresponding  findings  of  weaknesses  reported  which  caused 
the  team  to  make  that  judgment.  The  team  should  be  prepared  to  discuss  the 
data  upon  which  the  findings  were  based,  while  adhering  to  previously  agreed 
to  confidentiality  and  non-attribution  principles. 

Prepare  final  findings  presentation.  This  culminates  the  on-site  activities.  At 
this  point,  all  team  judgments  have  been  made.  This  activity  is  designed  to 
prepare  the  results  to  the  sponsor  and  organization  in  a  way  that  is  most 
beneficial  to  meeting  the  original  evaluation  goals. 

Much  of  the  preparation  of  the  briefing  could  be  started  in  advance  of  the  site 
visit,  using  a  template.  The  template  is  then  filled  in  with  actual  data  (findings, 
ratings,  data  about  the  appraisal  itself)  during  this  step.  At  this  point  there 
should  be  no  debate  about  the  findings  or  ratings.  The  primary  discussion  is 
about  how  the  judgments  made  by  the  team  will  be  presented  to  best  achieve 
evaluation  goals. 

Present  final  findings.  The  material  is  presented  in  a  stand  up  briefing  format. 
Typically,  the  team  leader  will  be  the  sole  presenter,  with  team  members 
supporting  the  team  leader  during  questions  and  answers  with  the  participants. 
The  professionalism  with  which  the  team  has  carried  out  its  activities 
throughout  the  week  will  be  evidenced  by  the  manner  in  which  the  participants 
at  the  findings  briefing  react  to  the  results. 

At  this  briefing,  it  is  highly  encouraged  to  have  as  many  people  from  the 
organization  participate  as  possible,  not  just  the  people  who  were  directly 
involved  in  the  evaluation.  During  this  briefing  the  sponsor  or  chosen  delegate 
will  reemphasize  and  renew  his  or  her  commitment  to  the  evaluation  process, 
and  to  the  use  of  the  results. 

Final  findings  presentation  content.  The  final  findings  presentation  will  contain 
the  appraisal  results  in  accordance  with  the  evaluation  plan.  This  always 
means  reporting  the  findings  of  strengths,  weaknesses,  and  improvement 
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Outputs 


activities  in  each  KPA  investigated.  KPA  profiles  are  also  reported  for  any 
model  components  that  were  formally  rated  (including  maturity  level  if  this 
option  is  selected).  Global  findings  and  non-reference  model  findings  are  also 
provided.  Besides  findings  and  ratings,  the  team  should  remind  the  participants 
about  the 


•  method  used  to  generate  the  results 

•  team  qualifications  and  sponsorship  for  the  evaluation 

•  amount  of  labor  that  was  expended  during  the  appraisal 

•  breadth  and  depth  of  the  organization  evaluated  (number  of  instruments, 
projects,  interviewees,  and  documents  reviewed) 

•  next  steps  —  how  the  results  will  be  used,  by  whom,  and  when 

Close  out  site  activities.  During  the  final  period  on  site,  the  team,  or  perhaps 
only  the  team  leader,  will  privately  discuss  with  the  sponsor  any  questions  or 
concerns  he  or  she  has  about  the  evaluation  or  its  results.  Confidentiality  and 
non-attribution  principles  agreed  to  with  the  sponsor  during  planning  are  still  in 
effect  during  this  meeting.  Discussion  of  next  steps  may  occur  at  this  time.  The 
sponsor  may  also  choose  to  have  a  select  group  of  senior  managers 
participate  in  this  meeting. 

Following  the  meeting  with  the  sponsor,  the  team  leader  will  work  with  the  team 
in  making  assignments  for  generating  the  various  reports  that  will  be  needed. 
Creating  the  reports  is  ultimately  the  team  leader’s  responsibility,  but  is  a 
shared  tasking  with  the  other  team  members.  The  team  may  concurrently 
begin  closing  out  logistical  aspects  of  the  site  visit  when  the  team  leader  is 
meeting  with  the  sponsor.  It  is  important  to  make  team  assignments  and  the 
projected  schedule  for  follow  on  activities  that  team  members  may  be  involved 
in  while  on  site.  It  is  easy  to  lose  momentum  after  an  appraisal.  Taking  on 
specific  actions  while  on  site  will  keep  the  momentum  going. 

Final  findings  presentation,  including 

•  global  findings 

•  final  findings 

•  non-reference  model  (e.g.,  non-CMM)  findings 

•  ratings 

•  meta  data  about  the  evaluation  itself 


no 
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Outcome 


Options 


The  sponsor  and  the  appraised  organization  understand  and  accept  the  team’s 
findings. 

This  activity  may  occur  at  different  times  in  some  applications.  The  default 
option  is  to  deliver  the  final  findings  at  the  close  of  the  site  visit.  This  is  preferred 
for  facilitating  continuous  process  improvement  goals.  However,  the  delivery  of 
the  final  findings  to  the  appraised  organization  may  be  limited  by  legal  and 
contractual  constraints  (such  as  in  government  source  selection).  Findings 
should  always  be  delivered  at  the  earliest  possible  time  within  these 
constraints. 

Additionally,  depending  on  how  "^appraisal  constraints  are  factored  into  the 
evaluation  plan,  global  findings  and  non-reference  model  findings  may  not  be 
generated  to  save  time  throughout  the  site  visit. 

Presentation  of  the  final  findings  is  usually  done  with  all  participants  and  others 
in  the  organization  at  one  time,  but  the  organization  may  choose  to  have  the 
team  present  the  findings  at  multiple  times  during  the  last  site  visit  day. 

Meeting  with  the  sponsor  on  the  last  site  visit  day  is  not  feasible  in  an 
acquisition  context,  since  the  sponsor  is  not  from  the  appraised  organization. 
The  senior  site  manager  should  be  invited  to  the  executive  session. 
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3.15  Activity  15  Produce  Reports  and  Support  Follow-On 
Activities 

Table  3-19  and  Figure  3-17  below  provide  an  overview  of  the  steps  in  this 
activity. 


1  Activity 

1  Steps 

1  Outcome  1 

Produce  Reports 

Step  15A:  Produce  Reports 

A  formal  baseline  of  the  appraisal  conduct 

and  Support 

-  Findings  Report 

and  results  is  established  and  reports  are 

Follow-On 

-  Outcomes  Report 

delivered  to  stakeholders. 

Activities 

-  Evaluation  Data  Report 

-  Method  Evaluation  Report 

Step  1 5B:  Distribute  Reports 

Step  1 5C:  Preserve  and/or  Dispose  Of 
Records 

Step  15D:  Support  Follow-On  Activities 

The  evaluation  results  are  used  to  support 
business  objectives. 

Table  3-19:  Produce  Reports  and  Support  Follow-On  Activities 


Purpose  Produce  a  formal  baseline  of  the  appraisal  conduct  and  results  for  the  sponsor 

and  other  stakeholders,  and  ensure  the  evaluation  results  are  available  to 
achieve  stated  business  objectives. 

Inputs  Evaluation  artifacts 

•  evaluation  plan 

•  site  information 

•  all  presentations 

•  all  annotated  worksheets/checklists 

•  all  working  notes 

Action  This  activity  will  require  follow-through  and  commitment  by  the  evaluation 

team.  Execution  of  the  activity  will  vary  depending  on  the  use  of  the  appraisal 
outcomes. 

Produce  ^appraisal  reports.  Several  reports  are  generated.  Each  is  briefly 
discussed  below. 

The  findings  (sometimes  called  “final”)  report  is  an  essential  item  for  closing  out 
an  evaluation,  because  it  defines  the  baseline  of  all  activities  and  results  from 
the  team’s  execution  of  the  method.  This  baseline  is  used  for  subsequent 
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Figure  3-17:  Produce  Reports  and  Support  Follow-On  Activities  Activity  Diagram 

reports,  and  also  for  use  in  subsequent  appraisals  (in  all  cases  except  a  source 
selection  SCE).  The  report  should  be  completed  as  soon  after  the  on  site 
period  as  possible.  Usually  this  should  be  no  more  than  two  weeks. 


The  findings  report  should  contain  the  following  information: 


•  Evaluation  goals,  requirements,  and  scope. 

•  Information  common  to  all  development  organization(s),  includes  the 
Product  Profiles,  organization  charts  and  other  site  information,  and 
questionnaire  responses. 

•  All  worksheets  and  checklists. 

•  Objective  evidence  which  serves  as  a  basis  for  findings.  (This  section 
should  be  a  formal  description  of  the  evidence  supporting  the  team’s 
findings  rather  than  the  actual  evidence.  The  team  will  not  be  allowed  to 
take  the  evidence  with  them.) 
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•  Findings,  including  a  separate  sheet  for  each  KPA.  The  findings  sheets 
should  include  references  to  the  objective  evidence  which  support  them. 

•  Ratings  (for  all  model  components  rated). 

•  Risks  associated  with  the  accuracy  and  completeness  of  evaluation 
outputs. 

An  outcomes  report  includes  recommendations  for  use  of  the  evaluation 
results,  in  accordance  with  the  planned  use  of  the  results  defined  in  Activities 
1  and  2.  In  many  applications,  recommendations  for  use  of  the  results  will  be 
combined  with  the  findings  report  information  in  one  document  called  the  “final” 
report.  The  outcomes  report  documents  the  official  baseline  of  what  was  done, 
and  the  recommended  use  of  the  results.  The  report  should  be  completed  as 
soon  after  the  on  site  period  as  possible.  Recommendations,  depending  on 
their  nature,  complexity,  and  use,  can  take  skilled  teams  as  much  as  two 
months  to  produce,  depending  on  the  level  of  effort  allocated. 

An  appraisal  data  report  is  a  summary  of  essential  characteristics  and  results 
of  the  evaluation  that  can  be  submitted  to  bodies  responsible  for  qualifying 
appraisers,  or  reporting  state  of  the  practice  information  to  the  community.  An 
example  of  an  institution  supporting  this  role  for  the  software  community  is  the 
SEI.  Software  process  appraisal  data  may  be  submitted  directly  to  the  SEI  or 
through  a  cognizant  sponsor  institution  (e.g.,  an  acquisition  agency). 

The  appraisal  data  report  also  documents  items  important  to  the  qualification 
of  individuals  on  the  team  by  formalizing: 

•  who  was  on  the  team 

•  when  they  were  trained 

•  what  method  they  executed 

•  what  the  results  were 

A  method  evaluation  report  is  a  compilation  of  lessons  learned  and 
suggestions  for  improvement  of  the  method  used  by  the  team,  which  is  sent  to 
the  SEI. 

Typically,  the  templates  for  all  of  these  reports  are  provided  to  individuals  as 
part  of  evaluator  training.  The  findings  and  outcomes  report  may  be  tailored  by 
the  user.  The  appraisal  data  and  method  evaluation  reports  are  standard. 

Distribute  reports  and  dispose  of  data.  After  the  various  reports  are 
delivered  to  the  appropriate  stakeholders,  the  team  leader  is  responsible  for 
properly  disposing  of  the  evaluation  data,  in  accordance  with  agreements 
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Outputs 


Outcome 


made  with  the  sponsor  during  Activities  1  and  2.  How  the  records  will  be 
preserved  or  disposed  is  dependent  on  the  application  of  the  method  and  its 
appraisal  goals. 

In  all  applications  of  SCE,  strict  non-attribution  policies  apply.  However, 
confidentiality  rules  may  differ  by  application.  In  an  SCE  for  acquisition,  the 
results  are  not  “confidential”  in  that  the  sponsor  is  an  outside  organization  from 
the  recipient.  But  the  results  are  only  known  to  the  sponsor  and  the  recipient. 
Competing  organizations  do  not  see  the  results. 

The  recipient  organization,  if  the  sponsor  agrees  and  it  is  planned  for,  may 
always  choose  to  make  the  results  known  outside  the  organization.  At  a  high 
level,  this  might  be  done  for  marketing  and  public  relations  reasons. 

Support  follow-on  activities.  Follow  on  activities  are  the  tasks  that  the  team 
leader  and  various  team  members  might  take  to  support  use  of  the  evaluation 
results.  This  is  highly  desirable  to  sponsors,  due  to  the  in  depth,  extensive  site 
information  that  is  gleaned  by  the  team  in  a  short  time  period.  This  knowledge 
can  be  put  to  use  in  many  ways: 

•  assisting  an  evaluation  board  in  identifying  offeror  risk 

•  assisting  an  award  fee  board  in  determining  incentives  to  a  contractor 

•  assisting  the  development  organization  in  producing  an  improvement  plan 

•  assisting  action  teams  in  planning  specific  process  improvements 

Evaluation  reports 

•  findings 

•  outcomes 

•  appraisal  data 

•  method  evaluation 

Evaluation  artifacts 
Evaluation  records  disposition 
Follow-up  actions 

A  formal  baseline  of  the  appraisal  conduct  and  results  is  delivered  to 
stakeholders. 

The  evaluation  results  are  used  to  support  business  objectives. 
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Options 


A  single,  final  report  may  include  both  the  findings  report  information  and  the 
outcomes  report  information. 

An  organization  may  choose  to  make  the  results  more  broadly  known  and 
available  than  is  standard,  given  adherence  to  non-attribution  policies. 

The  form  of  follow  on  support  varies  depending  on  the  intended  use  of  the 
results,  (e.g.,  risk  identification  in  acquisition,  award  fee  determination  in 
process  monitoring,  action  planning  in  internal  evaluations). 
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3.16  Summary  of  Inputs  and  Outputs 

The  table  below  summarizes  the  inputs  and  outputs  of  each  activity. 


Activity 


Outputs 


1 .  Analyze 

Requirements 


Evaluation  requirements 

•  business  context 

•  sponsor  objectives 

•  specific  sponsor  req’ts 
Resource  constraints 

•  personnel,  facility,  and  project  availability 

•  budget  availability 
Logistical  constraints 

•  program  plans 

•  external  schedule 

•  geographic  constraints 


Evaluation  requirements: 

•  evaluation  goals 

•  evaluation  constraints 
-  product  profile 
Evaluation  scope: 

•  reference  model  scope, 

•  organizational  scope, 

List  of  evaluation  outputs 
Sponsor  commitment 


2.  Develop  Evaluation  goals,  Initial  evaluation  plan, 

Evaluation  Plan  Evaluation  constraints,  Revised  evaluation  plan 

Reference  model  scope, 

Organizational  scope, 

List  of  appraisal  outputs, 

Team  leader  selection 
Site  information 


3.  Select  and  Evaluation  constraints, 

Prepare  Team  Product  profile, 

Reference  model  scope, 
Organizational  scope, 
Evaluation  plan 


4.  Obtain  Product  profile, 

Organizationali  Reference  model  scope, 

nformation  Organizational  scope, 

Evaluation  plan 


Team  leader  selection, 

Team  member  selection, 

Prepared  team1 

•  method  training 

•  reference  model  training 

•  team  building 

•  evaluation  orientation 

Request  for  organization  information, 
Site  information  (from  organization) 


Table  3-20:  SCE  V3.0  Activities  and  Their  Primary  Inputs  and  Outputs 


1.  Training  in  the  appraisal  method  and  demonstrated  knowledge  of  the  reference  model  are  requirements  for 
preparing  the  team. 

Note:  Throughout  the  document  and  in  these  tables,  several  assumptions  are  made  to  simplify  discussion. 
Some  items  are  consistent  throughout  all  the  activities  and  steps,  and  therefore  are  not  repeated  in  each 
section.  If  an  item  generally  affects  all  of  the  activities  and  steps,  it  is  listed  below. 

•  The  CMM  reference  model  is  an  input. 

•  Reference  model  knowledge  is  a  constraint. 

•  Teamwork  and  interpersonal  skills  are  a  constraint. 

•  Method  rules  (for  rating,  coverage,  etc.)  are  a  constraint. 

•  Items  completed  during  Activity  1  and  2  (reference  model  scope,  organizational  scope,  budget,  plan,  etc.) 
automatically  become  constraints  on  the  rest  of  the  activities. 
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1  Activity 

I  Inputs 

Outputs 

5.  Select  and 
Prepare 
Participants 

Evaluation  plan, 

Site  information, 

Profile  and  questionnaire  analyses 

Selected  site(s), 

Selected  project(s), 

Selected  interviewees, 

Initial  briefing, 

Prepared  participants 

6.  Prepare  For 

Data  Collection 

Evaluation  plan, 

Organization  charts  (from  site 
information), 

Profile  and  questionnaire  analyses. 
Selected  site(s), 

Selected  project(s), 

Selected  interviewees 

Data  collection  strategy 

•  interview 

•  document  review 

Data  collection  tactics 

•  interview  questions 

•  initial  document  request  list 

•  roles/responsibilities 

7.  Receive 

Presentations 

Evaluation  data 

•  development  organization  presentation 

•  site  information 

•  appraisal  schedule 

Updated  evaluation  data 

•  site  information 

•  appraisal  schedule 

•  terminology 

•  presentation  slides 

Working  notes 

Requests  for  additional  data 

8.  Review 
Documents 

Evaluation  data 

•  site  information 

•  initial  document  set 

•  annotated  worksheets/checklists 
Document  review  strategy 

Evaluation  data 
•  working  notes 

Requests  for  additional  data  (e.g., 
documents) 

9.  Conduct 
Interviews 

Evaluation  data 

•  appraisal  schedule 

•  site  information 

•  interview  questions 

•  working  notes 

•  annotated  worksheets/checklists 

•  interview  strategy 

Requests  for  documents 

Evaluation  data 
•  working  notes 

Requests  for  additional  data  (e.g., 
documents,  additional  or  new 
interviewees) 

10.  Consolidate 

Data 

Evaluation  data 

•  working  notes 

•  annotated  worksheets/checklists 

•  annotated  draft  findings  presentation 

Evaluation  data 

•  observations  (model  and  non-model) 

•  revised  data  collection  plan 

•  annotated  worksheets/checklists 

Requests  for  additional  data  (interviewees 
or  documents) 

11.  Deliver  Draft 
Findings 

Evaluation  data 

•  annotated  worksheets/checklists 

•  observations 

Evaluation  data 
•  working  notes 

Draft  findings  presentation 

12.  Make  Rating 
Judgments 

Evaluation  data 

•  annotated  worksheets/checklists 

•  working  notes 

Evaluation  outputs 

•  ratings  of  reference  model  components 
Evaluation  data 

•  annotated  worksheets/checklists 

Table  3-20:  SCE  V3.0  Activities  and  Their  Primary  Inputs  and  Outputs  (cont.) 
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13.  Deliver  Final  Evaluation  data  Final  findings  presentation 

Findings  •  annotated  worksheets/checklists  •  global  findings 

•  ratings  •  final  findings 

•  non-model  findings 

•  ratings 


14.  Produce 

Evaluation  artifacts 

Evaluation  reports 

Reports  and 

•  appraisal  plan 

•  findings 

Support  Follow- 

•  site  information 

•  outcomes 

On  Activities 

•  all  presentations 

•  appraisal  data 

•  Initial  briefing(s) 

•  method  evaluation 

•  Organization 

Evaluation  artifacts 

•  Draft  findings 

Evaluation  records  disposition 

•  Final  findings 

Follow-up  actions 

•  all  annotated  worksheets/checklists 

•  all  working  notes 

Table  3-20:  SCE  V3.0  Activities  and  Their  Primary  Inputs  and  Outputs  (cont.) 
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SCE  V 2.0  to  SCE  V3.0  Activity  Mapping 


Appendix  A  SCE  V2.0  to  SCE  V3.0  Activity 

Mapping 


1  V2.0  Phase 

V2.0  Step 

V2.0  Purpose 

V3.0  Activity  Map-  1 
ping  1 

Phase  1 : 

Evaluation 

Start 

1 .  Develop 

Target 

Product  Profile 

Understand  attributes  of  the  software  product  and 
the  project  required  to  produce  it. 

Activity  1 ,  Step  1 B, 
Determine 

Appraisal 

Constraints 

2.  Determine 
Target 

Process 

Capability 

Determine  the  process  capability  that  is  most 
appropriate  for  the  planned  development — the 
Target  Process  Capability. 

Activity  1,  Step  1C, 
Determine 

Reference  Model 
Scope 

3.  Select  Team 

Have  a  trained  team  in  place  to  execute  the  SCE. 

Activity  3,  Select 
and  Prepare  Team 

Phase  2: 

General 

Preparation 

4.  Create 
Experience 
Table 

Identify  areas  where  the  development 
organizations  lack  experience,  indicating  a 
potential  for  risk. 

Not  in  default 
method;  it  is  a 
source  selection 
specific  step.  Will 
occur  in  source 
selection  tailoring 
as  part  of  Activity  5, 
Analyze 

Instrument  Data. 

5.  Create  Critical 
Subprocess 
Area  List 

Define  and  document  the  scope  of  the  SCE,  in 
terms  of  critical  subprocess  areas  within  the 

Target  Process  Capability  KPAs. 

Deleted,  concept 
done  partially  in 
Activity  1 ,  Step  1 C, 
Reference  Model 
Scope.  Not  done  as 
part  of 

organizational 
profile  analysis. 
Selection  is  now 
based  directly  on 
goals. 

6.  Originate 
Validation 
Worksheets 

Record  the  set  of  critical  subprocess  areas  for  all 
development  organizations  on  forms  that  can  be 
used  in  subsequent  information  collection  efforts. 

Deleted, 
annotating 
worksheets 
subsumed  within 
various  steps. 

Table  A-1: 

SCE  V2.0  to  SCE  V3.0  Activity  Mapping 
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V2.0  Phase 

V2.0  Step 

V2.0  Purpose 

V3.0  Activity  Map-  1 
ping  1 

Phase  3: 

Specific 

Preparation 

7.  Select  Projects 
to  Investigate 

Select  projects  for  evaluation  that  give  the  most 
insight  into  the  processes  that  will  be  used. 

Activity  6,  Step  6B, 
Select  Projects 

8.  Develop  Key 
Issue 

Worksheet 

Create  a  consolidated  list  of  key  issues  for 
investigation  at  the  development  organization  site. 

Activity  7,  Step  7A, 
Prioritize  Focus 

Areas 

9.  Develop  Topic 
Lists 

Select  topics  for  probing  the  process 
implementation;  topics  define  observable  work 
practices  that  map  to  the  critical  subprocess 
areas. 

Activity  7,  Steps 

7A,  7B,  7D, 

Prepare  for  Data 
Collection 

10.  Add  Topics  to 
Validation 
Worksheet 

Capture  the  consolidated  topic  list  for  use  at  a 
particular  site. 

Deleted, 
annotating 
worksheets 
subsumed  within 
various  steps. 

1 1 .  Prepare  for 
Exploratory 
Interviews 

Develop  detailed  interview  strategy,  including  the 
team’s  decisions  on  who  will  be  interviewed,  when 
they  will  be  interviewed,  and  what  they  will  be 
asked. 

Activity  7,  Steps 

7B,  7C,  Prepare  for 
Data  Collection 

12.  Prepare  Entry 
Briefing 

Establish  the  agenda  for  the  initial  organization 
meeting  and  set  initial  expectations  for  the  site 
visit. 

Activity  6,  Step  6D, 
Prepare  Initial 
Briefing(s) 

Table  A-t :  SCE  V2.0  to  SCE  V3.0  Activity  Mapping  (cont.) 
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V2.0  Phase 

V2.0  Step 

V2.0  Purpose 

V3.0  Activity  Map¬ 
ping 

Phase  4: 

Site  Data 
Collection 

13.  Conduct  Initial 
Organization 
Meeting 

Clarify  expectations  of  the  SCE  site  visit. 

Activity  6,  Step  6E, 
Conduct  Initial 
Briefing(s) 

14.  Conduct  Initial 
Document 
Review 

Determine  the  degree  to  which  the  organization 
and  project-level  documentation  define  and 
support  standard  processes  for  the  KPAs  and 
subprocess  areas  under  investigation. 

Activity  9,  Review 
Documents 

15.  Conduct 
Exploratory 
Interviews 

Provide  insight  into  how  the  subprocess  areas  are 
implemented  in  practice,  determine  the  extent  that 
processes  have  been  internalized  by  the 
development  organizations,  identify  critical 
implementation-level  documents. 

Activity  1 0, 

Conduct  Interviews 

16.  Hold  Team 
Caucus 

Analyze,  share,  and  consolidate  information  in 
order  to  reach  conclusions  about  topics. 

Activity  1 1 ,  Step 

11  A,  Organize  and 
Combine  Data 

17.  Conduct 
Document 
Review 

Search  for  objective  evidence  of  how  processes 
are  implemented  at  the  working  level. 

Activity  9,  Review 
Documents 

18.  Develop 
Preliminary 
Findings 

Articulate  conclusions  about  the  subprocess 
areas  based  on  the  information  available,  guide 
subsequent  information-gathering  efforts. 

Activity  1 1 ,  Step 

1 1 B,  Determine 

Data  Sufficiency 

19.  Create 

Consolidation 

Plan 

Plan  and  initiate  further  data  collection. 

Activity  1 1 ,  Step 

11C,  Review  and 
Revise  Data 
Gathering  Plans 

20.  Conduct 
Consolidation 
Interviews 

Clarify  any  remaining  issues  by  confirming  or 
negating  candidate  findings  through  further 
interviews. 

Activity  1 0, 

Conduct  Interviews 

21.  Conduct  Final 
Document 
Review 

Clarify  any  remaining  issues  by  confirming  or 
negating  candidate  findings  through  further 
document  review. 

Activity  9,  Review 
Documents 

Table  A-1 :  SCE  V2.0  to  SCE  V3.0  Activity  Mapping  (cont.) 
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1  V2.0  Phase 

V2.0  Step 

V2.0  Purpose 

V3.0  Activity  Map-  1 
ping  I 

Phase  5: 
Findings 

22.  Determine 
Findings 

Validate  the  preliminary  findings  and  consolidate 
them  by  KPA. 

Activity  1 1 , 
Consolidate  Data, 
and  Activity  14, 

Step  14A,  Prepare 
Final  Findings 
Presentation 

23.  Produce 
Findings 

Report 

Document  the  SCE  activities  and  provide  a  formal 
record  of  the  findings. 

Activity  1 5,  Steps 

15A,  15B,  15C, 
Produce  Reports 
and  Support 
Follow-on  Activities 

24.  Conduct  Exit 
Briefing 

Provide  feedback  to  the  recipient  and  conclude 
the  SCE. 

Activity  1 4,  Step 

14C,  Close  Out 

Site  Activities 

Table  A-1:  SCE  V2.0  to  SCE  V3.0  Activity  Mapping  (cont.) 


Notes:  In  SCE  V2.0,  there  were  no  formal,  explicit  steps  for 

1 .  Analyzing  all  parts  of  the  requirements  activity  (SCE  V3.0  Activity  1  A,  1 E,  1 F); 

these  components  of  were  described  in  sections  of  the 
implementation  guide. 

2.  Documenting  the  appraisal  plan  (SCE  V3.0  Activity  2);  components  of  it  were 

described  in  sections  of  the  implementation  guide. 

3.  Tailoring  method  use  (SCE  V3.0  Activity  2E),  because  the  method  was 

designed  to  be  application  (source  selection)  specific. 

4.  Obtaining  organization  information  (SCE  V3.0  Activity  4),  as  it  was  implicit  in 

SCE  V2.0  Steps  4,  7,  and  8,  and  was  explicitly  described  in  the 
implementation  guide  (source  selection  specific). 

5.  Receiving  presentations  (SCE  V3.0  Activity  8);  this  was  described  in 

guidance  for  executing  SCE  V2.0  Step  13. 

6.  Delivering  draft  findings  (SCE  V3.0  Activity  12),  due  to  source  selection 

limitations. 

7.  Making  rating  judgments  (SCE  V3.0  Activity  13)  since  Maturity  Level 

determination  was  not  an  option. 

8.  Presenting  final  findings  (SCE  V3.0  Activity  14B),  due  to  source  selection 

considerations. 

9.  Supporting  follow-on  activities  (SCE  V3.0  Activity  15D),  as  prior  method 

rationale  was  to  strictly  separate  determination  of  appraisal  results 
from  the  use  of  those  results. 
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Appendix  B  SCE  V3.0  to  CAF  VI. 0  Requirements 

Traceability  Table 

SCE  V3.0  Method  High-Level  Architecture 

The  diagram  below  depicts  the  high  level  architecture  of  the  SCE  V3.0  Method.  This 
architecture  is  adapted  from  the  SEI  CMM  Appraisal  Framework  (CAF)  [Masters  95], 


Figure  B-1 :  SCE  V3.0  High-Level  Architecture 
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The  principal  differences  between  this  figure  and  the  analogous  figure  in  the  CAF  are 

•  follow-up  actions  are  shown  as  an  output  of  Report  Results 

•  appraisal  team  leader  provides  leadership  as  an  input  to  the  appraisal  team 

•  appraisal  participants  are  depicted  on  the  figure  only  once 

•  the  appraisal  plan  is  not  shown  as  a  direct  input  to  Report  Results 

•  appraisal  outputs  specify  both  findings  and  ratings 

1  Phase  Activity 

Steps 

CAF  VI  .0  Req’ts 

Plan  and  1 .  Analyze 

1A  Determine  Appraisal  Goals 

R3 

Prepare  Requirements 

IB  Determine  Appraisal  Constraints 

R3 

Appraisal 

1C  Determine  Reference  Model  Scope 

R4 

ID  Determine  Organizational  Scope 

R5 

1 E  Determine  Appraisal  Outputs 

None 

1 F  Obtain  Commitment 

R6 

2.  Develop 

2A  Identify  Required  Resources 

R15 

Appraisal  Plan 

2B  Identify  Cost 

R15 

2C  Identify  Schedule 

R15,  R16 

2D  Work  Out  Logistics 

R17 

2E  Tailor  Method 

R15 

2F  Plan  Use  of  Appraisal  Outputs 

R15 

3.  Select  and 

3A  Select  Team  Leader 

R7,  R8 

Prepare  Team 

3B  Select  Team  Members 

R7,  R8,  R9 

3C  Prepare  Team 

RIO 

4.  Obtain 

4A  Identify  Required  Information 

None 

Organizational 

4B  Request  Information 

R19 

Information 

4C  Provide  Information  (Organization) 

None 

5.  Analyze 

5A  Receive  and  Summarize  Instrument  Data 

R19 

Instrument 

5B  Examine  and  Review  Instrument  Data 

R19 

Data 

5C  Develop  Profile(s)  Analyses 

R19 

6.  Select  and 

6A  Select  Site(s) 

R11 

Prepare 

6B  Select  Project(s) 

R12 

Participants 

6C  Select  Participants 

R13 

6D  Prepare  Initial  Briefing(s) 

None 

6E  Conduct  Initial  Briefing(s) 

R14 

7.  Prepare  For 

7 A  Prioritize  Focus  Areas 

R19 

Data 

7B  Establish  Interview  Strategy 

R19 

Collection 

7C  Script  Questions 

R19 

7D  Establish  Document  Review  Strategy 

R19 

7E  Revise  Appraisal  Team  Roles  and 

R19 

Responsibilities 

Table  B-1:  SCE  Activities  and  Their  Associated  Steps  and  CAF  Requirements 
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Phase 

Activity 

Steps 

CAF  VI  .0  Req’ts 

Conduct  8.  Receive  8 A  Listen  R19 

Appraisal  Presentations  8B  Ask  Questions  R19 

8C  Take  and  Tag  Notes  R19 


Conduct 

9.  Review 

9A  Determine  Information  Needed 

R19 

Appraisal 

Documents 

9B  Select  or  Request  Documents 

R19 

(cont.) 

9C  Review  Documents 

R19 

9D  Take  and  Tag  Notes 

R19 

10.  Conduct 

10A  Determine  Information  Needed 

R19 

Interviews 

10B  Select  or  Request  Interviewees 

R19 

10C  Ask  Questions 

R19 

1 0D  Listen 

R19 

10E  Take  and  Tag  Notes 

R19 

11.  Consolidate 

11 A  Organize  and  Combine  Data 

R20 

Data 

11 B  Determine  Data  Sufficiency 

R21,  R22 

11C  Review  and  Revise  Data  Gathering  Plan 

R23,  R24,  R25 

12.  Deliver  Draft 

12A  Prepare  Draft  Findings  Presentation 

R19,  R21 

Findings 

12B  Present  Draft  Findings  and  Solicit  Feedback 

R19,  R21 

12C  Listen 

R19,  R21 

12D  Take  and  Tag  Notes 

R19,  R21 

13.  Make  Rating 

13A  Judge  Satisfaction  of  Key  Practices 

R30,  R31,  R32 

Judgments 

13B  Judge  Satisfaction  of  Common  Feature 

R30,  R31,  R32 

13C  Judge  Satisfaction  of  the  Process  Area  Goals 

R26,  R28,  R29, 

Based  on  Implementation  and 

R30,  R31 ,  R32, 

Institutionalization 

R33 

13D  Judge  Satisfaction  of  Process  Areas 

R26,  R28,  R29, 

R30,  R31,  R32, 

R34 

13E  Determine  Maturity  Level 

R27,  R29,  R31, 

R32,  R35 

Report 

14.  Deliver  Final 

1 4A  Prepare  Final  Findings  Presentation 

R36 

Appraisal 

Findings 

14B  Present  Final  Findings 

R37 

Results 

1 4C  Close  Out  Site  Activities 

None 

15.  Produce 

1 5A  Product  Reports 

R36,  R37 

Reports  and 

1 5B  Distribute  Reports 

R38 

Support 

1 5C  Preserve  and/or  Dispose  of  Records 

R39,  R40 

Follow-On 

Activities 

1 5D  Support  Follow-On  Activities 

None 

Table  B-1:  SCE  Activities  and  Their  Associated  Steps  and  CAF  Requirements  (cont.) 

Note:  CAF  VI. 0  requirements  R1-R2  (document  the  method,  provide  guidance  for  three 
phases  of  appraisal  execution),  and  R18  (select  artifacts  to  support  method  activities)  are 
satisfied  by  creating  the  SCE  V3.0  materials  (Method  Description,  Implementation  Guide,  and 
Training). 
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April  1996 


SCE  V 3.0  Temporal  Flow 


Appendix  C 


SCE  V3.0  Temporal  Flow 
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SCE  V3.0  Temporal  Flow 


i 


Day 

Time 

SCE  V3.0  Activity 

Parallel  Activities 

Calendar  Time  and 

Allotted 

Notes 

Pre 

8  hrs 

1 .  Analyze  Requirements 

1  week 

On-site 

16  hrs 

2.  Develop  Evaluation  Plan 

Select  and  Prepare 
Team 

2  weeks 

40  hrs 

3.  Select  and  Prepare  Team 

Obtain 

4  weeks  —  assumes 

Organizational 

Information 

team  training  required; 
does  not  include  travel 

4  hrs 

4.  Obtain  Organizational 

Select  and  Prepare 

4  weeks  (assumes  an 

Information 

Team 

acquisition  SCE) 

8  hrs 

5.  Analyze  Instrument  Data 

Select  and  Prepare 
Participants 

1  week 

16  hrs 

6.  Select  and  Prepare  Participants 

Prepare  for  Data 
Collection 

2  weeks  —  time  does 
not  include  any  travel 

16  hrs 

7.  Prepare  for  Data  Collection 

Assumes  multiple 
projects 

On  Site 

1  hr 

8.  Receive  Presentations 

Initial  Briefing  from 

Day  1 

2  hrs 

9.  Review  Documents 

Ac6  may  be  done  here 
Initial  document  review 
only  —  may  be  done 
during  pre  on-site 

4  hrs 

10.  Conduct  Interviews 

All-on-one 

management 
interviews  typically 

3  hrs 

1 1 .  Consolidate  Data 

Day  2 

6  hrs 

10.  Conduct  Interviews 

All-on-many 
(practitioner) 
interviews  typically 

2  hrs 

9.  Review  Documents 

3  hrs 

11.  Consolidate  Data 

Review  Documents 

Day  3 

12.  Deliver  Draft  Findings 

Bf  ^ 

9,  10.  Conduct  Interviews  and 

Both  done  in  parallel 

Few-on-one  interviews 

Review  Documents 

typically 

2  hrs 

1 1 .  Consolidate  Information 

Review  Documents, 

Data  collection  is  very 

Conduct  Interviews 

focused 

3  hrs 

13.  Make  Rating  Judgments 

Day  4 

4  hrs 

14.  Deliver  Final  Findings 

Post  On- 

32  hrs 

15.  Produce  Reports  and  Support 

8  weeks  —  Time 

Site 

Follow-On  Activities 

allotted  is  minimum 

Sub- 

108  hrs 

Plan  and  Prepare  Phase 

Includes  training  as  a 

Total 

one  time  major  event 

Sub- 

32  hrs 

Conduct  Evaluation  Phase 

Total 

Sub- 

36  hrs 

Report  Evaluation  Results  Phase 

Follow  on  activities 

Follow  on  time  is 

Total 

additional 

Table  C-1:  Temporal  Flow  of  SCE  V3.0  Activities 
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Day 

Time 

SCE  V3.0  Activity 

Parallel  Activities 

Calendar  Time  and 

Allotted 

Notes 

Grand 

Total 

176  hrs 

Table  C-1:  Temporal  Flow  of  SCE  V3.0  Activities  (cont.) 
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Appendix  D  SCE  Information  Flow 

D.1  Introduction 

The  following  figures  depict  the  SCE  V3.0  method  from  an  information  flow  perspective.  The 
diagrams  were  generated  using  a  simple  data  flow  diagramming  tool.  Due  to  features  of  the 
tool  used,  several  points  should  be  made  in  regard  to  the  way  information  is  depicted. 

Notes 

In  order  to  improve  readability  of  the  diagrams,  a  few  of  the  data  flows  are  not  depicted.  For 
example,  the  data  flow  between  Develop  Evaluation  Plan  and  Select  and  Prepare  Participants 
showing  the  Evaluation  Plan  as  an  output  from  the  former  and  an  input  to  the  latter  is  not 
shown.  This  input/output  relationship  is  shown  in  the  SCE  V3.0  Activities  Table  preceding  the 
flow  diagrams. 

The  names  of  the  processes  do  not  always  match  the  names  of  the  method  activities  shown 
in  the  Activities  Table  due  to  the  character  number  limitation  of  the  DFD  tool  used.  An  example 
is  that  the  “Plan  and  Prepare  for  Evaluation”  phase  is  simply  described  as  “Plan  and  Prepare” 
in  the  flow  diagram. 

Data  Reports,  generated  from  the  DFD  tool  describing  the  information  flow  items,  are  provided 
in  an  appendix.  In  this  version  of  the  method  description,  the  reports  only  describe  processes, 
data  stores,  and  external  entities.  The  reader  may  wish  to  review  these  reports  during  or  after 
looking  at  the  information  flows,  because  they  contain  textual  information  about  each  of  the 
items. 

D.2  How  To  Read  These  Information  Flows 

The  information  flow  diagrams  can  be  read  much  like  a  standard  data  flow  diagram  (DFD). 
The  specific  conventions  embedded  in  the  DFD  tool  used  to  create  the  diagrams  are  listed 
below. 
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SCE  Information  Flow 


April  1996 


External  entities,  Data  stores,  Processes,  and  Data  flows 


External  entities  are  shown  as  a  box  with  a  shaded  background. 

Data  stores  are  shown  a  rectangle  with  an  open  side.  Data  stores  are  defined  to  allow  the 
appropriate  data  flow  inputs  and  outputs  to  processes. 

Processes  are  shown  as  boxes  with  rounded  corners.  Processes  in  these  diagrams  are 
equivalent  to  method  activities  in  the  phase  diagrams. 

Data  flows  are  depicted  by  the  lines  going  between  processes,  external  entities,  or  data 
stores.  They  can  be  either  unidirectional  or  bidirectional. 
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Inherited  item,  in-coming  flow  and  Inherited  item,  out-going  flow 


An  item  with  a  small  box  and  arrow  on  the  top  of  it  depicts  a  process,  external  entity,  or  data 
store  inherited  from  a  parent  diagram.  An  arrow  going  “out”  of  the  small  box  on  top  of  the  item 
depicts  an  inherited  item  with  an  out-going  flow.  An  arrow  going  “in”  to  the  small  box  depicts 
an  inherited  item  with  an  in-coming  flow.  The  name  of  the  inherited  item  is  exactly  as  stated 
on  the  parent  item.  The  inherited  items  are  used  for  consistency  checking  between  parent  and 
child  diagrams  in  the  DFD  structure. 
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Figure  D-1:  SCE  V3.0  System  Diagram 
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Appendix  E  SCE  V3.0  Version  Description 

The  SCE  method  is  designed  to  resolve  outstanding  community  issues  with  the  SEi  appraisal 
methods.  Summary  level  community  requirements,  collected  by  the  SEI  during  the  1 992-1 993 
time  frame,  can  be  summarized  in  the  following  broad  areas: 

•  Baseline  the  SEI  appraisal  methods 

•  Make  the  methods  public 

•  Incorporate  the  CMM  into  the  SEI  methods 

•  Align  assessment  and  evaluation  methods  with  each  other 

The  first  three  bullets  were  addressed  by  previous  versions  of  the  SCE  method.  Evolution  of 
SCE  V2.0  to  SCE  V3.0  principally  effects  the  fourth  bullet,  aligning  SEI  methods  by  incorpo¬ 
rating  the  CMM  Appraisal  Framework  (CAF).  The  CAF  provides  essential  requirements  for 
any  method  which  chooses  to  be  CMM-based.  SCE  V3.0  continues  to  be  baselined,  public, 
and  make  use  of  the  published  CMM. 

Impact  of  major  changes 

The  changes  described  above  have  made  it  easier  to  tailor  an  SCE  to  the  business  needs  of 
the  sponsor.  They  improve  the  utility  and  versatility  of  the  method  by  providing  more  thorough 
and  detailed  guidance  to  users  of  the  method.  Finally,  the  changes  continue  to  provide  a  base¬ 
line  for  orderly  public  evolution  of  the  method. 

The  major  changes  from  SCE  V2.0  to  V3.0  are: 

•  CMM  Appraisal  Framework  (CAF)  compliance 

•  implementation  of  requirements  beyond  minimum  CAF  compliance 

•  support  follow  on  activities 

•  rate  KPAs  (if  selected  in  planning) 

•  plan  use  of  appraisal  outputs 

•  CMM  knowledge  is  a  prerequisite 

•  method  independent  of  the  application 

•  evaluation  process  improvement  —  reduction  in  the  number  of  method  activities 

•  Plan  and  Prepare  For  Evaluation  phase  changes 

•  experience  table  used  in  acquisition  applications  only 

•  elimination  of  subprocess  areas  —  reference  model  boundaries  determined  directly 
from  goals  or  other  model  components 

•  Conduct  Evaluation  phase  changes  —  collect  and  record  data 

•  group  interview  techniques  added 
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•  presentations  as  a  data  collection  tool  added 

•  instruments  may  provide  a  source  of  observations 

•  explicit  examination  of  key  practices  (when  selected  in  planning) 

•  explicit  examination  of  alternative  practices 

•  Conduct  Evaluation  phase  changes  —  making  judgments 
.  explicit  model  coverage  and  judgment  rules  added 

•  non-reference  model  findings  are  captured 

•  Report  Evaluation  Results  phase  changes 

•  formal  ratings  have  been  reinstated  and  rating  rules  are  explicit 

•  ratings  are  provided  for  any  model  component  that  meets  method  rules  and  was 
explicitly  planned  for 

.  maturity  level  rating  is  reinstated  as  an  option 

.  follow  on  activities  have  been  added  (support  the  use  of  appraisal  results) 

•  Application  changes 

•  method  is  extendable  for  use  with  other  models  (such  as  the  Trusted  Systems  CMM) 

•  contractor  teaming  arrangements  are  addressed 

CAF  Compliance 

The  SCE  method  V3.0  fully  implements  the  requirements  of  the  CAF.  The  CAF  defines  re¬ 
quirements  for  method  developers  to  implement  should  they  choose  to  build  a  CMM-based 
method.  The  CAF  is  a  critical  component  of  the  SEI  strategy  for  meeting  the  customer  com¬ 
munity  requirement  of  aligning  the  appraisal  methods.  A  SCE  V3.0/  CAF  requirements  trace- 
ability  table  is  provided  in  Appendix  B. 

Implementation  of  requirements  beyond  minimum  CAF  compliance 

•  support  follow  on  activities 

The  CAF  requirements  end  with  the  delivery  of  appraisal  reports  and  the  proper  disposition 
of  the  appraisal  data.  SCE  V3.0  includes  activities  supporting  use  of  the  appraisal  results 
to  help  meet  business  objectives.  Relative  to  SCE  V2.0,  this  specifically  means  that  one 
or  more  SCE  team  members  are  expected  to  assist  the  procurement  officials  in 
determining  risk  assessments  using  the  SCE  findings.  Users  have  routinely  asked  that  this 
important  aspect  be  addressed  in  the  SCE  method  documentation.  Principally,  this  comes 
in  the  form  of  additional  guidance  in  the  SCE  V3.0  Implementation  Guide  for  Acquisition. 
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•  rate  key  practices  and  common  features  (if  selected  in  planning) 

The  CAF  requirements  for  rating  CMM  components  addresses  goals,  KPAs,  and  maturity 
level  (if  selected  as  an  option).  SCE  V3.0  includes  rating  key  practices  and  common 
features  of  the  CMM  if  these  components  are  selected  during  appraisal  planning,  and  the 
data  collected  by  the  team  meets  the  method’s  coverage  and  corroboration  rules.  This 
addresses  user  needs  of  increasing  the  specificity  of  appraisal  results  to  better  support 
follow  on  activities  such  as  action  planning  and  risk  assessment. 

•  plan  use  of  appraisal  outputs 

The  CAF  has  no  requirements  directly  related  to  planning  the  use  of  appraisal  outputs. 
This  is  related  to  the  CAF  not  requiring  support  of  follow  on  activities.  In  SCE,  this  planning 
has  always  been  an  important  aspect,  since  the  government  regulations  enforcing  source 
selection  procedures  dictate  that  this  planning  occur  up  front,  prior  to  conducting  a  site 
visit.  In  SCE  V3.0,  planning  the  use  of  appraisal  outputs  up  front  in  the  appraisal  process 
will  continue  to  be  supported  by  the  method. 

In  general,  the  rigorous  process  that  users  have  come  to  expect  in  executing  the  SCE  method 
will  continue. 

CMM  knowledge  is  a  prerequisite 

User  feedback  over  the  past  three  years  has  made  it  very  clear  that  successful  implementation 
of  the  SCE  method  is  highly  dependent  on  the  CMM  knowledge  of  the  team  members.  Lack 
of  this  knowledge  has  been  an  important  reason  of  perceived  inconsistencies  in  SCE  results, 
and  for  increasing  difficulties  in  properly  executing  the  method.  Much  of  this  relates  to  the 
sheer  size  and  depth  of  the  CMM.  CMM  knowledge  has  always  been  strongly  recommended. 
Requiring  CMM  knowledge,  coupled  with  improved  and  expanded  CMM  training  associated 
with  evaluator  training,  will  help  achieve  the  community  repeatability  and  consistency  goals. 

Method  independent  of  its  application 

The  preparatory  steps  conducted  in  previous  version  of  the  SCE  Method  focused  each  SCE 
on  the  software  processes  that  contribute  the  most  to  risk  for  the  planned  development.  When 
the  method  was  question  based,  SCE  teams  looked  at  essentially  the  same  software  process¬ 
es  each  time  the  method  was  used.  By  emphasizing  the  KPAs  and  subprocess  areas,  the 
SCE  VI  .5  and  V2.0  methods  allowed  a  team  to  select  the  areas  for  evaluation  that  are  most 
important  for  the  given  use  of  the  method.  This  tied  the  method  directly  to  its  application,  but 
limited  its  evolution  to  other  uses  and  the  comparability  of  results  across  applications. 

In  the  design  of  SCE  V3.0,  the  application  of  the  method  has  been  separately  documented, 
this  allows  greater  flexibility  in  the  baseline  method,  while  continuing  to  provide  the  implemen¬ 
tation  detail  which  was  greatly  valued  by  the  original  sponsors  of  the  method.  This  separation 
is  principally  achieved  by  improving  the  SCE  V3.0  Implementation  Guide  for  Acquisition. 
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Evaluation  process  improvement  —  reduction  of  the  number  of  activities 

Both  SCE  VI  .5  and  V2.0  have  the  same  24  discrete  steps,  divided  into  5  phases.  Document¬ 
ing  the  “as  is”  appraisal  process  used  in  the  SCE  method  during  the  period  between  1991- 
1994  was  an  important  step  towards  satisfying  customer  concerns  about  public  documenta¬ 
tion,  and  created  a  sound  basis  for  process  improvement  in  the  method. 

In  SCE  V3.0,  the  original  24  steps  and  5  phases  have  been  reduced  to  15  activities  and  3 
phases.  This  was  accomplished  by 

•  mapping  SCE  phases  directly  to  the  CAF  phases 

•  deleting  redundant  or  “low-level”  steps 

•  recombining  several  steps  with  similar  functions 

•  adding  9  implicit  or  missing  high  level  activities  and  detailed  steps 

The  result  of  these  improvements  is  a  net  37%  reduction  in  overall  process  activities  while  si¬ 
multaneously  increasing  the  completeness  of  appraisal  process  coverage.  Thus,  the  method 
is  more  intellectually  manageable,  allowing  teams  to  focus  more  time  on  the  technical  chal¬ 
lenges  of  collecting  and  making  judgments  about  process  data,  rather  than  on  the  execution 
of  a  complex  set  of  steps. 

Plan  and  Prepare  For  Evaluation  phase  changes 

•  experience  table  used  in  acquisition  applications  only 

This  change  is  linked  to  the  goal  of  separating  the  baseline  method  from  its  associated 
applications.  The  previous  published  versions  of  SCE  were  tightly  coupled  to  meeting  the 
needs  of  the  original  method  sponsor  (the  U.S.  DoD)  for  a  specific  application  —  source 
selection.  This  coupling,  although  extremely  valuable  to  the  sponsor,  limited  and 
constrained  innovative  uses  of  the  method  for  process  monitoring  or  internal  evaluation. 
Since  the  experience  table  is  principally  used  by  teams  in  the  general  preparation  activities 
designed  to  set  a  “level  playing  field”  in  an  acquisition,  this  aspect  could  be  specified  in  the 
method  supplement  for  acquisition,  rather  than  in  the  baseline  SCE  V3.0  method. 

•  elimination  of  subprocess  areas  —  reference  model  boundaries  are  determined  directly 
from  goals  or  other  model  components 

The  KPAs  in  the  maturity  model  were  refined  to  include  subprocess  areas  in  an  earlier 
version  of  SCE.1  Also,  a  set  of  elements  2  (now  called  features)  common  to  all  of  the 
subprocess  areas  was  defined  to  help  the  teams  select  topics  to  be  evaluated.  This  was 
necessary  at  the  time  because  of  the  shift  away  from  a  question-based  method  to  a  model 
based  method,  and  because  it  was  implemented  prior  to  the  release  of  the  CMM. 


1  In  version  2.0  of  the  SCE  Method,  the  subprocess  areas  are  derived  from  the  goals  of  CMM  vl.i  [Paulk  93a]. 
Previous  versions  of  the  SCE  Method  were  not  CMM-based. 

2  Previous  versions  of  SCE  used  the  term  element.  In  version  2.0  of  SCE,  the  term  features  was  used  in  place 
of  elements.  The  term  has  changed;  the  concept  has  not. 
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Collectively,  subprocess  areas  and  their  common  elements  improved  the  SCE  team’s 
ability  to  probe  specific  software  processes.  As  the  method  and  model  evolved,  and  the 
SEI  intent  to  align  the  appraisal  methods  became  clear,  the  original  notion  of  subprocess 
areas  changed  to  be  directly  mapped  to  the  CMM  VI. 1.  The  subprocess  areas  in  SCE 
V2.0  were  defined  by  a  one-to-one  relationship  to  the  goals  of  the  CMM  VI. 1  KPAs.  As 
such,  the  value  added  to  the  method  no  longer  outweighed  the  added  burden  of  terms, 
forms,  steps,  etc.  The  SCE  V3.0  method  was  simplified  by  tying  activities  for  determining 
reference  model  scope  directly  to  the  CMM,  eliminating  the  need  for  subprocess  areas. 

Conduct  Evaluation  phase  changes  —  collect  and  record  data 

•  group  interview  techniques  added 

Previous  versions  of  SCE  defined  interviews  as  solely  a  “many-on-one”  activity.  This 
satisfied  procurement  officials  desire  for  “unbiased”  input.  However,  recipient 
organizations  have  for  years  complained  about  the  “intimidation  factor”  of  this  process  on 
junior  level  practitioners,  and  have  requested  others  to  be  present  in  the  room  to  avoid 
“mistakes.”  This  was  especially  important  to  recipients  in  a  source  selection  context. 

Furthermore,  the  SEI  Software  Process  Assessment  (SPA)  method  solely  used  group 
interviews  (“many-on-many”),  called  Functional  Area  Representatives  (FARs),  to  obtain 
data  from  practitioners. 

SCE  V3.0  has  addressed  industry  concerns  about  the  many-on-one  SCE  interview  and 
inconsistencies  with  assessment  methods  by  adding  group  interviews  to  the  baseline 
method.  Allowing  groups  of  related  practitioners  to  be  interviewed  together  potentially 
increases  the  amount  and  breadth  of  data  collected,  reduces  the  intimidation  factor,  and 
is  consistent  with  CBA  I  PI. 

Options  for  conducting  practitioner  interviews  in  a  many-on-one  mode  have  been  retained 
to  allow  flexibility  for  sponsors.  Management  interviews  may  also  be  done  in  a  many-on- 
many  mode  in  addition  to  the  standard  many-on-one  style. 

•  presentations  as  a  data  collection  tool  added 

The  baseline  SCE  V3.0  method  explicitly  adds  the  use  of  presentations  to  collect  and 
validate  data.  Thus,  organizational  presentations  to  the  team  may  generate  observations, 
and  presentations  by  the  team  to  the  participants  may  also  generate  data  as  well  as 
validate  preliminary  findings.  Options  in  acquisition  for  minimizing  team  interaction  with  the 
site  participants  in  feedback  sessions  has  been  retained. 

•  instruments  may  provide  a  source  of  observations 

In  the  original  SCE  method  [Humphrey  87b],  the  questionnaire  essentially  defined  the 
method.  In  SCE  VI. 5  and  V2.0  [SCE  93,  SCE  94],  the  maturity  questionnaire  was  only 
used  as  a  starting  point  by  the  team  in  focusing  effort  and  prioritizing  time  on  site.  Data 
from  the  questionnaire  could  not  be  used  as  part  of  the  consolidation  and  judgment 
process.  This  was  due  in  part  to  the  limitations  inherent  in  previous  versions  of  the 
questionnaire,  the  method,  and  the  model. 

In  SCE  V3.0,  instrument  data  can  be  used  to  generate  observations  used  during  the 
consolidation  and  judgment  activities.  Instrument  data  includes  the  CMM  Maturity 
Questionnaire  VI  .1 ,  and  the  SCE  profiles.  There  are  specific  rules  for  how  these  items  can 
be  used. 
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•  explicit  examination  of  key  practices  (when  selected  in  planning) 

Although  SCE  teams  have  routinely  used  the  entire  CMM  since  it  was  published, 
previously  published  versions  of  the  SCE  method  did  not  explicitly  encourage  direct 
examination  of  key  practices.  This  was  based  on  a  principle  well  documented  in  the  CMM 
and  supported  by  industry  that  the  CMM  key  practices  are  not  intended  to  be  prescriptive. 
There  was  a  general  concern  that  SCE  teams  examining  key  practices  would  create  an 
environment  that  was  too  “audit-like.” 

User  and  recipient  feedback  over  time  has  strongly  suggested  that  although  this  is  a  valid 
concern,  it  belies  how  people  actually  use  the  CMM  as  a  reference  model.  SCE  V3.0 
allows  explicit  examination  of  key  practices  when  these  items  are  selected  by  the  sponsor 
and  the  team  during  appraisal  planning.  Rating  of  these  items  cannot  occur  unless  method 
rules  have  been  followed  (see  rating  of  key  practices  and  common  features  in  this  section). 
Coupled  with  improved  training  and  explicit  examination  of  alternative  practices,  the  risks 
cited  above  are  minimized. 

•  explicit  examination  of  alternative  practices 

SCE  V3.0  has  added  explicit  examination  of  alternative  practices  to  those  in  the  CMM. 
This  activity  is  embedded  within  the  method  and  associated  training.  This  supports 
concepts  in  the  CMM,  allowing  organization  greater  flexibility  when  concerned  about 
externally  led  evaluations,  and  allows  SCE  teams  to  examine  processes  in  greater  detail 
and  report  results  with  much  more  specificity  than  in  the  past. 

Conduct  Evaluation  phase  changes  —  making  judgments 

•  explicit  model  coverage  and  judgment  rules  added 

SCE  V3.0  has  added  explicit  model  coverage  and  judgment  rules.  These  concepts  were 
supported  by  previous  SCE  versions,  but  were  not  as  explicit  as  necessary  to  fully  support 
repeatability  and  consistency  goals.  The  CAF  requirements  are  embedded  in  the  SCE 
V3.0  method.  Method  documentation  provides  additional  guidance. 

•  non-reference  model  findings  are  captured 

Previous  versions  of  SCE  limited  teams  to  collecting  data  and  reporting  results  only  in 
those  pre-defined  CMM  components  identified  in  planning.  SCE  V3.0  has  made  collection 
and  reporting  of  non-reference  model  findings  a  part  of  the  baseline  method.  Thus, 
information  that  comes  up  during  a  site  visit  that  may  be  important  to  the  recipient 
organization  and  the  sponsor,  but  is  not  directly  related  to  the  model,  is  not  lost. 

Report  Evaluation  Results  phase  changes 

•  formal  ratings  have  been  reinstated  and  rating  rules  are  explicit 

Earlier  versions  of  the  method  only  reported  the  appraisal  findings  (strengths  and 
weaknesses).  In  SCE  V3.0,  ratings  rules  have  been  added  such  that  model  component 
satisfaction  ratings  can  be  produced  by  the  team.  This  helps  achieve  the  goal  of  improving 
communication  of  appraisal  results  between  the  team  and  their  sponsors. 
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•  ratings  are  provide  for  any  model  component  that  meets  method  rules  and  was  explicitly 
planned  for  during  the  plan  and  prepare  appraisal  phase 

Method  rules  now  require  the  team  to  rate  a  model  component  if  it  is  planned  to  be  rated, 
and  the  team  is  able  to  meet  related  method  rules  for  coverage  and  validation  during  the 
site  visit.  Assuming  these  rules  are  met,  teams  shall  rate  the  associated  component.  This 
helps  achieve  the  goal  of  providing  more  specific  actionable  results  to  the  sponsor. 

•  maturity  level  rating  is  reinstated  as  an  option 

Maturity  level  ratings  can  be  useful  to  describe  improvement  goals,  or  for  process 
improvement  efforts,  or  for  describing  capability  to  customers.  In  earlier  versions  of  the 
SCE  method,  the  method  and  model  documentation  was  not  sufficiently  robust  to  allow 
teams  to  repeatedly  and  consistently  determine  maturity  ratings.  The  key  issue  was  in 
translating  the  detailed  findings  of  strengths  and  weaknesses  into  a  “satisfaction” 
determination.  User  feedback  has  shown  that  both  types  of  information  are  needed  to 
support  sponsor  business  goals  and  process  improvement  goals.  These  considerations, 
coupled  with  improvements  in  the  method,  its  associated  documentation  and  training,  and 
its  reference  model  have  allowed  the  method  developers  to  reinstate  maturity  level  ratings 
as  a  viable  option  to  users.  This  is  the  first  time  since  the  release  of  the  original  Maturity 
Questionnaire  [Humphrey  87b]  that  there  has  been  an  explicit  scoring  mechanism  built 
into  the  method. 

•  follow  on  activities  have  been  added  (support  use  of  appraisal  results) 

This  activity  is  a  key  link  between  the  appraisal  process  and  the  system  level  process  that 
uses  the  method  results.  Users  and  recipient  organizations  have  requested  more 
information  on  translating  findings  into  useful  outcomes,  such  as  risk  identification  in  an 
acquisitions.  SCE  V3.0  adds  guidance  for  this  activity  in  the  baseline  method. 

Application  changes 

•  contractor  teaming  arrangements  are  addressed 

In  the  1 980’s,  use  of  the  SCE  method  focused  on  one  site,  principally  because  the  majority 
of  the  system  and  software  was  typically  created  and  integrated  at  one  location.  With 
increasing  use  of  contractor  teams  and  subcontractors  in  the  1990’s  to  produce  important 
portions  of  the  system  and  software,  SCE  V3.0  includes  a  discussion  of  conducting  SCEs 
in  this  environment.  This  discussion  can  be  found  in  the  SCE  Version  3.0  Implementation 
Guide  for  Supplier  Selection. 

•  method  is  extendable  for  use  with  other  models  (such  as  the  Trusted  Systems  CMM) 

An  essential  design  characteristic  of  SCE  V3.0  was  to  prepare  a  “generic”  appraisal 
process  that  would  maintain  fidelity  to  the  CMM  for  Software  VI. 1,  but  would  also  be 
extendable  to  the  various  new  models  that  have  recently  been  created  or  are  in 
development.  SCE  V3.0  builds  in  the  ability  to  readily  evolve  to  these  models  without 
requiring  sponsors  and  users  to  build,  learn,  and  maintain  additional  appraisal  methods. 
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Appendix  F  Attribute  Definitions 

This  appendix  contains  definitions  of  the  attributes  used  in  the  SCE  method. 
The  attributes  specify  important  characteristics  of  a  product  or  environment  for 
developing  a  product  that  significantly  impact  the  processes  that  are 
implemented  by  the  organization. 

The  attributes  allow  an  SCE  team  to  make  comparisons  between  profiles  in  a 
systematic  way.  Attributes  are  used  to  compare  previous  development 
organization  and  end  user  experience  to  the  attributes  of  the  current,  target,  or 
desired  development.  This  comparison  identifies  potential  risk  areas  that  help 
to  refine  and  focus  the  SCE. 

The  same  attributes  are  used  to  populate  all  of  the  profiles  used  in  the  method. 
These  profiles  are  created  and  used  during  the  Plan  and  Prepare  for  Appraisal 
phase.  The  profiles  are  an  “instrument”  data  collection  mechanism. 

F.1  Profile  Attributes 

Application  domain 

The  application  domain  attribute  indicates  the  area  of  subject  matter  expertise. 

There  is  no  accepted  taxonomy  of  application  domains;  however,  the  concept 
is  widely  understood  and  used.  Information  systems,  command  and  control 
systems,  weapon  systems,  simulation  systems,  training  systems,  avionic 
systems,  sensing  systems,  and  so  on  are  all  recognized  and  accepted  as 
different  application  domains.  What  makes  application  domains  different  is  the 
operational  environment  that  uses  the  system.  The  unique  characteristics  of 
the  operational  environment  are 

•  The  mission  for  which  the  system  is  needed. 

•  The  roles  and  responsibilities  of  the  people  who  interface  with  the 
system. 

•  The  resources  that  the  system  depends  upon,  which  defines  the 
potential  limit  of  the  services  that  the  system  can  provide  the  people 
in  the  operational  environment. 

Product  type 

The  product  type  attribute  refers  to  the  particular  aspect  of  the  application 
domain  which  the  system  will  support  or  to  the  type  of  service  which  the  system 
will  provide.  It  may  be  considered  a  subset  of  the  application  domain. 
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For  example,  communications  or  displays  could  be  product  types  in  a 
command  and  control  system,  a  weapons  system  or  other  application  domain. 
Although  there  may  be  similarities  in  the  communications  subsystem  in  the 
various  application  domains,  they  each  have  their  own  set  of  unique  problems 
that  must  be  addressed. 


Size 

The  size  attribute  is  composed  of  three  related  attributes. 

•  duration 

•  team  size 

•  estimated  size 

The  duration  is  the  estimated  or  required  length  of  time  to  deliver  the  product. 
It  is  usually  specified  in  number  of  months.  The  team  size  is  the  number  of 
engineers  (e.g.,  software  developers)  who  will  be  involved  in  the  project.  This 
includes  both  direct  roles  (e.g.,  design)  and  support  roles  (e.g.,  quality 
assurance).  The  estimated  size  is  the  base  element  used  in  planning  an  effort. 
The  parameter  used  depends  on  the  system  being  built  and  the  purpose  of  the 
appraisal.  In  a  software  system,  the  size  would  be  stated  in  source  lines  of 
code,  number  of  function  points,  or  some  similar  measure  suited  to  the 
development. 

There  is  no  standard  way  of  measuring  the  size  attributes.  For  an  SCE,  the 
specific  method  used  is  not  as  important  as  consistently  using  whatever 
method  is  chosen  so  comparisons  will  be  meaningful. 

Reuse  estimate 

The  reuse  estimate  attribute  indicates  the  development  organization’s 
approach  to  building  the  product.  It  is  correlated  with  the  size  attribute.  Reuse 
estimates  are  indicated  by  the  percentage  of  the  product  that  is 

•  new 

•  modified  (from  existing  components) 

•  reused  (previous  components  used  as  is) 

Processes  implemented  for  accomplishing  high  amounts  of  reuse  are 
significantly  different  from  processes  for  developing  new  items. 


Type  of  work 

The  type  of  work  attribute  is  used  to  indicate  the  portion  of  the  life  cycle  which 
will  be  performed  by  the  development  organization. 
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The  following  are  examples  of  different  types  of  work  that  may  be  required: 

•  full  development:  The  development  organization  is  required  to  build 
a  product  based  upon  the  system  requirements.  The  development 
organization  will  typically  be  required  to  complete  software 
requirements,  top  level  design,  detailed  design,  code  and  unit  test, 
and  acceptance  testing  at  the  development  organization’s  site. 

•  code  development:  The  development  organization  is  required  to 
develop  code  according  to  the  system  requirements  and  software  top 
level  design  provided  by  the  issuing  authority.  This  type  of 
development  might  be  done  under  a  delivery  order  contract.  The 
development  organization  may  do  the  detailed  design,  coding, 
integration,  and  testing,  but  the  system  testing  may  be  done  by  the 
customer. 

•  system  development  without  coding:  The  development  organization 
may  be  required  to  do  all  the  work  except  the  software  detailed 
design  and  development. 

•  a  prime  contract  acquisition:  In  a  large  system  acquisition  there  may 
be  many  organizations  who  subcontract  significant  parts  of  the 
system,  especially  software  parts.  The  prime  contractor  allocates 
system  requirements  to  the  subcontractor,  integrates  the 
components,  and  conducts  acceptance  tests. 

Development  team  approach 

The  development  team  approach  attribute  is  related  to  how  the  developer 
organizes  itself  to  produce  the  system;  the  degree  to  which  various 
development  and  support  groups,  and  the  customer,  interact  and  are  brought 
to  bear  on  the  effort.  This  attribute  is  indicated  by  various  development 
approach  terms,  such  as: 

•  integrated  product  teams 

•  multi-functional  teams 

•  interdisciplinary 

•  standard  functional 


Processes  implemented  will  differ  significantly  between  standard  functional 
and  integrated  product  team  approaches.  This  is  not  intended  to  be  an  all 
inclusive  list.  There  is  no  industry  standard  taxonomy  for  describing  the  various 
approaches  to  development.  Again,  consistency  in  the  terminology  used  is 
most  important. 
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Language(s) 

The  language  attribute  indicates  the  programming  languages  in  which  the 
software  code  is  written  (or  will  be  written).  If  multiple  languages  are  used  or 
expected,  showing  the  languages  and  percent  of  the  system  implemented  in 
them  is  useful. 

Customer 

The  customer  attribute  indicates  who  the  development  is  being  done  for. 
Examples  include  a: 

•  defense  agency 

•  federal  agency 

•  commercial  market 

•  commercial  client 

•  foreign  government 

•  company  internal  organization 

Applicable  standards 

The  applicable  standards  attribute  indicates  the  standards  that  are  imposed  on 
the  project  that  produces  the  product,  such  as  ISO-9000-3,  MIL-STD-498, 
DoD-STD-2167A,  DoD-STD-2168,  or  draft  MIL-STD-499.  Standards  often 
reflect  the  development  and  operational  “rigor”  which  is  required  by  customers 
in  specified  application  domains,  product  types,  and  types  of  work.  They  could 
include  buyer  specific  standards,  emerging  industry  standards  or  reference 
models,  or  development  organization  specific  standards. 

Major  Subcontractors 

The  major  subcontractors  attribute  is  used  to  indicate  whether  the  development 
organization  has  or  plans  to  have  outside  sources  produce  a  substantial 
amount,  primary  components,  or  mission  critical  aspects  of  the  system 
functionality.  If  there  are  no  major  subcontractors,  this  item  is  not  applicable  to 
the  evaluation.  The  major  subcontractors  attribute  does  not  replace  the 
subcontract  management  process  area  of  various  reference  models. 


Precedence 

The  precedence  attribute  indicates  whether  the  developer,  end  user,  and 
customer  (if  different  from  the  end  user)  have  experience  with  the  planned, 
target,  or  desired  system  and  the  environment  for  producing  that  system.  The 
values  for  this  attribute  are  “no”  (meaning  the  entity  creating  the  profile  believes 
the  system  is  unprecedented),  or  “yes”  (meaning  the  system  is  precedented  to 
the  entity  creating  the  profile.)  Systems  that  are  providing  a  new  capability  tend 
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to  have  more  requirements  changes  that  exert  stress  on  existing  process 
capability.  Also,  if  known,  a  percentage  of  the  system  deemed  to  be 
unprecedented  is  useful  information. 


Target 

The  target  attribute  indicates  the  hardware,  software,  and  telecommunications 
environment  that  the  developed  system  will  operate  in. 

0.1  Other  Related  Items 

Status  and  environment  information  on  the  current  projects  submitted  for  evaluation  is  asked 
for  by  the  sponsor  on  another  instrument.  The  profiles  which  use  these  attributes  are  used  by 
the  team  during  the  project  selection  step  during  preparation  activities.  Status  and  environ¬ 
ment  items  are  listed  below. 

Schedule 

The  schedule  attribute  is  pertinent  to  current  work  efforts  reflected  in  the 
Product  Profiles  submitted  for  evaluation  by  the  recipient.  They  identify  where 
the  development  organization  is  on  each  project. 

•  Start 

The  start  attribute  shows  when  the  project  actually  began. 


•  Current  month 

The  current  month  attribute  is  the  number  of  months  since  the  start  of  the 
project. 


•  Current  phase 

The  current  phase  attribute  refers  to  the  life  cycle  phase  of  the  development 
which  the  project  is  currently  in,  such  as  requirements  analysis,  design,  coding, 
unit,  integration,  or  acceptance  testing. 


•  Requirements  ends 

The  requirements  ends  attribute  shows  how  long  after  the  start  of  the  project 
the  requirements  phase  was  completed  or  is  scheduled  to  be  completed. 


•  Design  ends 

The  design  ends  attribute  shows  how  long  after  the  start  of  the  project  the 
design  phase  was  completed  or  is  scheduled  to  be  completed. 
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•  Coding  ends 

The  coding  ends  attribute  shows  how  long  after  the  start  of  the  project  the 
coding  phase  was  completed  or  is  expected  to  be  completed. 

•  Testing  ends 

The  testing  ends  attribute  shows  how  long  after  the  start  of  the  project  the 
testing  phase  was  completed  or  is  expected  to  be  completed. 


•  Schedule  adjustment 

The  schedule  adjustment  attribute  shows  how  many  months  the  schedule  has 
slipped,  or  been  formally  changed,  since  the  original  plan. 


Environment 

The  environment  attribute  refers  to  the  hardware,  software,  and 
telecommunications  environment  used  to  develop  the  system.  This  is  the 
systems  and  software  engineering  environment.  This  attribute  is  principally 
concerned  with  the  tools  component  of  the  environment  (rather  than  the  people 
or  process  aspects).  This  is  important  information  because  automated  tools 
differ  in  the  type  of  methodologies,  degrees  of  tool  integration,  levels  of  process 
enforcement,  and  amount  of  process  flexibility  they  support. 

A  list  of  major  tools  used  is  appropriate.  Important  information  sought  is: 

•  principal  development  methodologies  incorporated  in  the  tools 

•  degree  of  integration  between  tools 

•  level  of  process  enforcement  embedded  in  the  tools 

•  process  management  support  tools  used 

Terms  depicting  the  environment,  such  as  “integrated  CASE,”  may  also  be 
useful.  There  is  no  industry  standard  taxonomy  for  this  area,  although  there  are 
emerging  standards  that  can  be  referenced  which  discuss  these  items. 
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Appendix  G  Glossary 

Ability  to  perform:  One  of  five  common  features  in  the  CMM  for  Software.  The  ability  to  per¬ 
form  reflects  the  preconditions  that  must  exist  in  the  project  or  organization  to  implement  the 
software  process  competently.  Ability  to  Perform  typically  involves  the  features  of  resources, 
organization  structures,  and  training. 

Accuracy:  An  observation  is  considered  to  be  accurate  if  the  appraisal  team  agrees  that  it  is 
based  on  what  is  heard  and  seen,  is  worded  appropriately,  and  is  correctly  categorized  and 
classified  [Masters  95]. 

Activities  performed:  One  of  five  common  features  in  the  CMM  for  Software.  Activities  per¬ 
formed  describe  the  roles  and  procedures  necessary  to  implement  a  key  process  area.  Activ¬ 
ities  performed  typically  involves  the  features  of  establishing  plans  and  procedures, 
performing  the  work,  tracking  it,  and  taking  corrective  action. 

Activity:  A  key  practice  of  the  activities  performed  common  feature  in  the  CMM  for  Software. 

Acquisition:  The  cradle-to-grave  life  cycle  of  a  system  or  product,  and  one  of  the  primary  ap¬ 
plications  of  the  SCE  method.  When  used  during  the  pre-contract  award  phase  of  an  acquisi¬ 
tion,  may  be  called  a  source  selection  SCE,  in  reference  to  the  U.S.  Department  of  Defense 
(DoD)  term  for  the  process  of  selecting  a  supplier  in  an  acquisition.  When  used  during  the  con¬ 
tract  execution  phase,  may  be  called  a  process  monitoring  SCE.  The  purpose  of  a  supplier 
selection  SCE  is  to  provide  input  to  the  sponsor  on  the  process  capability  of  one  or  more  de¬ 
velopment  organizations.  The  outcome  from  a  supplier  selection  SCE  is  the  selection  of  the 
best  value  supplier  for  performance  of  a  planned  contract.  SCE  results  are  just  one  aspect 
considered  in  the  sponsor’s  decision.  (See  acquisition  agency  and  sponsoring  organization.) 

Acquisition  agency:  An  organization  responsible  for  developing,  delivering,  and  supporting 
a  system  or  product.  Not  normally  the  producer  of  the  product.  For  purposes  of  this  document, 
an  acquisition  agency  is  the  appraisal  sponsoring  organization  when  applying  the  SCE  meth¬ 
od  for  the  purpose  of  selecting  a  supplier.  (See  sponsoring  organization.) 

Alternative  practice:  Practices  which  are  implemented  differently  from  those  described  in  the 
reference  model  that  may  accomplish  the  goals  of  a  process  area. 

Anomaly:  A  contradictory  response  to  the  same  question  on  the  questionnaire,  or  from  other 
data  collection  mechanisms,  by  two  (or  more)  projects.  May  indicate  an  issue  that  needs  to  be 
probed  further.  Related  to  inconsistency. 

Applicable  standards:  An  attribute  used  in  SCE.  This  attribute  indicates  the  government  or 
commercial  development  and  quality  standards  that  are  imposed  on  the  project  or  organiza¬ 
tion,  such  as  DoD-STD-21 67A,  DoD-STD-2168,  MIL-STD-1521B,  or  MIL-STD-498,  or  ISO 
9000-3. 
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Application  of  the  SCE  method:  Synonym  for  use  of  the  SCE  method. 

Application  domain:  An  attribute  used  in  SCE.  An  application  domain  is  “a  bounded  set  of 
related  systems  (i.e.,  systems  that  address  a  particular  type  of  problem).  Development  and 
maintenance  in  an  application  domain  usually  require  special  skills  and/or  resources.  Exam¬ 
ples  include  payroll  and  personnel  systems,  command  and  control  systems,  compilers,  and 
expert  systems”  [Paulk  93b].  For  SCE,  this  is  an  attribute  used  within  the  various  profiles.  The 
application  domain  attribute  indicates  the  area  of  subject  matter  expertise  needed  to  translate 
system  requirements  into  software  requirements,  and  indicates  significant  differences  in  the 
engineering  practices  which  transform  the  software  requirements  into  accepted  code. 

Appraisal:  An  expert  or  official  valuation  of  something  [AHD  85].  In  the  context  of  model- 
based  process  appraisals,  an  appraisal  is  an  examination,  by  a  trained  team,  of  an  organiza¬ 
tion's  current  practices  from  a  process  management  perspective.  This  is  a  dynamic  concept 
—  the  act  of  appraising  (contrast  with  appraisal  method). 

Appraisal  constraints:  Constraints  that  affect  appraisal  conduct  such  as  budget  limitations, 
schedule  limitations,  and  resource  limitations  (people  and  facilities)  [Masters  95]. 

Appraisal  goals:  The  desired  outcome  of  an  appraisal  process  [Masters  95]. 

Appraisal  method:  The  documented  process  for  conducting  an  evaluation  or  assessment  of 
something.  Specific  to  SCE,  the  sequence  of  steps  performed  for  evaluating  the  process  ca¬ 
pability  of  an  organization  relative  to  a  reference  model.  Also,  a  set  of  activities,  tools,  and 
techniques  used  by  people  to  appraise  the  process  capability  of  an  organization  at  a  given 
point  in  time.  An  appraisal  method  describes  a  process  —  “a  sequence  of  steps  performed  for 
a  given  purpose”  [IEEE].  The  term  appraisal  method  typically  refers  to  the  method  itself,  but 
may  also  be  used  to  connote  the  method  and  its  associated  documentation  and  training  ma¬ 
terials. 

Appraisal  outputs:  Any  lasting  artifact  produced  by  the  team  in  executing  the  appraisal.  In 
SCE,  the  primary  output  from  the  site  visit  is  the  set  of  findings.  Often  synonymous  with  ap¬ 
praisal  results,  although  in  the  SCE  context  appraisal  outputs  is  a  broader  term,  because  re¬ 
sults  only  relate  to  the  findings  and  ratings  generated. 

Appraisal  reports:  The  set  of  documented  artifacts  created  by  the  appraisal  team  as  a  result 
of  conducting  an  appraisal.  These  reports  include:  findings  briefings  and  reports,  an  outcomes 
report,  an  appraisal  data  report,  and  a  method  evaluation  report.  Collectively,  they  form  the 
official  record,  or  baseline,  of  the  appraisal  for  subsequent  use  by  the  sponsor  or  other  stake¬ 
holders  in  the  data  and/or  process  executed.  All  reports  are  generated  after  the  conclusion  of 
the  site  visit  except  for  the  findings  briefing. 

Appraisal  requirements:  Appraisal  goals  and  constraints  [Masters  95]. 
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Appraisal  risk:  Risk  is  a  measure  of  uncertainty  of  attaining  a  goal,  objective,  or  requirement 
pertaining  to  technical  performance,  cost,  and  schedule.  Risk  level  is  categorized  by  the  prob¬ 
ability  of  occurrence  and  the  consequences  of  occurrence.  This  includes  the  adverse  conse¬ 
quences  of  process  variability  [MIL-STD-499B].  For  SCE,  appraisal  risk  has  two  components: 
technical  risk  inherent  in  the  method  as  defined  or  tailored,  and  process  risk  in  executing  the 
method.  Appraisal  risk  is  manifested  in  the  likelihood  (probability)  of  errors  in  the  results  (i.e., 
that  the  findings  and  ratings  are  incorrect).  (See  rating  baseline.) 

Appraisal  scope:  The  boundaries  of  the  investigation,  in  terms  of  the  breadth  within  the  or¬ 
ganization  and  the  depth  within  the  reference  model  used.  The  organizational  entities  and 
CMM  components  selected  for  investigation  [Masters  95].  (See  organizational  scope  and  ref¬ 
erence  model  scope.) 

Appraised  entity:  The  organizational  units  to  which  appraisal  outputs  apply.  An  appraised 
entity  may  be  any  portion  of  an  organization  including  an  entire  company,  a  selected  business 
unit,  a  specific  geographic  site,  units  supporting  a  particular  product  line,  units  involved  in  a 
particular  type  of  service,  an  individual  project,  or  a  multi-company  team  [Masters  95]. 

Artifact:  an  object  produced  or  shaped  by  human  workmanship  [AHD  85],  For  model  based 
process  appraisals,  artifacts  are  the  products  resulting  from  enacting  a  process. 

Attributes:  characteristics  of  a  software  product  or  project.  The  attributes  used  in  SCE  are 
defined  throughout  this  glossary  and  are  discussed  in  another  appendix  of  the  method  de¬ 
scription. 

Audit:  An  independent  examination  of  a  work  product  or  set  of  work  products  to  determine 
compliance  with  specifications,  standards,  contractual  agreements,  or  other  criteria  [Paulk 
93b], 

Candidate  findings:  Synonym  for  observations.  Candidate  findings  are  observations  for 
which  there  is  not  yet  enough  objective  evidence  to  make  a  decision  (an  unvalidated  obser¬ 
vation).  (See  observations.) 

Caucus:  A  meeting  in  which  the  team  analyzes  information  they  have  learned  while  on  site 
during  appraisal  conduct,  including  interviews,  document  review,  and  presentations,  to  trans¬ 
form  data  into  observations  and  finally  into  findings.  SCE  teams  routinely  participate  in  cau¬ 
cuses,  or  team  meetings,  during  an  SCE  site  visit.  These  caucuses  are  designed  to  help 
achieve  consensus  among  the  team  members.  SCE  team  members  analyze,  share,  and  con¬ 
solidate  information  in  order  to  reach  conclusions  about  what  was  seen  and  heard  as  a  result 
of  their  data  collection  activities.  (See  consolidation.) 
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Capability  Maturity  Model1  (CMM(SM)):  “A  description  of  the  stages  through  which  software 
organizations  evolve  as  they  define,  implement,  measure,  control,  and  improve  their  software 
processes”  [Paulk  93b],  For  SCE  this  is  a  model  which  is  used  to  evaluate  a  development  or¬ 
ganization’s  process  capability.  (See  maturity  model.) 

Commitment  to  perform:  One  of  five  common  features  in  the  CMM  for  Software.  Commit¬ 
ment  to  perform  reflects  the  actions  that  the  organization  must  take  to  ensure  that  the  process 
is  established  and  will  endure.  Commitment  to  perform  typically  involves  the  features  of  es¬ 
tablishing  organizational  policies  and  senior  management  sponsorship.  A  commitment  is  a 
pact  that  is  freely  assumed,  visible,  and  expected  to  be  kept  by  all  parties  [Paulk  93b]. 

Common  feature:  “An  attribute  that  indicates  whether  the  implementation  and  institutional¬ 
ization  of  a  key  practice  is  effective,  repeatable,  and  lasting”  [Paulk  93b],  There  are  five  com¬ 
mon  features  defined  for  CMM  vl.1:  commitment  to  perform,  ability  to  perform,  activities 
performed,  measurement  and  analysis,  and  verifying  implementation. 

Competitive  range:  Key  term  relating  to  the  acquisition  use  of  the  SCE  method  in  govern¬ 
ment  source  selection.  By  law  (10U.S.C.  2304  [g])  written  or  oral  discussions  in  negotiated 
procurements  must  be  conducted  with  all  responsible  offerors  who  submit  proposals  within  a 
competitive  range.  The  determination  as  to  which  proposals  are  not  in  the  competitive  range, 
and  the  exclusion  of  offerors  either  before  or  as  a  result  of  written  or  oral  discussions,  will  be 
made  by  the  Contracting  Officer,  subject  to  the  approval  of  the  sponsor.  The  sponsor  may  des¬ 
ignate  the  evaluation  team  chairperson  to  accomplish  this  function. 

The  competitive  range  must  be  determined  after  evaluation  of  all  proposals  received,  on  the 
basis  of  price  or  cost,  technical,  and  other  salient  factors  including  proposal  deficiencies  and 
their  potential  for  correction.  The  competitive  range  must  include  all  proposals  which  have  a 
reasonable  chance  of  being  selected.  The  objective  is  not  to  eliminate  proposals  from  the 
competitive  range,  but  to  facilitate  competition  by  conducting  written  and  oral  discussions  with 
all  offerors  who  have  a  reasonable  chance  of  being  selected  for  an  award  [USAF  84]. 

Consistency:  The  degree  of  uniformity,  standardization,  and  freedom  from  contradiction 
among  documents  or  system  components.  Consistency  of  an  appraisal  method  refers  to  the 
ability  of  different  appraisal  teams  using  the  same  method  to  conduct  appraisals  of  the  same 
scope  to  produce  non-conflicting  results  [Masters  95], 

Consolidation:  The  decision  making  activity  in  the  iterative  information  gathering,  organizing, 
and  analyzing  components  of  the  SCE  process.  The  activities  conducted  by  the  appraisal 
team  to  transform  raw  data  collected  from  the  recipient  organization  into  observations  and 
findings.  Consolidation  activities  occur  throughout  the  site  visit. 


1  Capability  Maturity  Model  and  CMM  are  service  marks  of  Carnegie  Mellon  University. 
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Contract  monitoring:  A  specific  application  of  the  SCE  method.  Euphemism  for  process 
monitoring.  Part  of  the  process  monitoring  “family”  of  evaluations.  (See  process  monitoring.) 

Corroboration:  In  SCE,  a  synonym  for  confirmation.  All  appraisal  observations  must  be  con¬ 
firmed  by  information  from  different  sources  and  different  data  gathering  sessions  prior  to  use 
as  findings.  This  is  sometimes  referred  to  in  the  SCE  method  as  rules  for  confirming  observa¬ 
tions. 

Coverage:  The  extent  to  which  data  gathered  fully  addresses  reference  model  components, 
organizational  units,  and  life  cycle  phases  within  the  scope  of  an  appraisal  [Masters  95].  For 
SCE,  the  link  between  coverage  and  rating  is  important.  One  or  more  validated  observations 
that  the  team  agrees  fully  cover  the  area  of  investigation  and  meet  method  rules  for  corrobo¬ 
ration  (multiple  sources,  multiple  sessions,  documentation,  etc.)  are  said  to  be  sufficient  for 
rating  the  reference  model  items.  (See  sufficiency  for  rating,  validation,  and  corroboration.) 

Customer:  An  attribute  in  SCE.  This  attribute  indicates  who  the  development  is  being  done 
for. 

Data:  Information,  especially  information  organized  for  analysis  or  used  as  the  basis  for  a  de¬ 
cision  [AHD  85]. 

Data  collection:  The  method  activities  related  to  obtaining  information  from  the  appraised  en¬ 
tity  for  the  purpose  of  evaluating  process  capability.  Four  data  sources  are  used  in  the  SCE 
method:  interviews,  document  review,  presentations,  and  instruments. 

Development  organization:  An  organization  that  develops  and/or  maintains  software  prod¬ 
ucts.  The  development  organization  is  the  recipient  of  an  SCE. 

Development  organization  community:  All  of  the  development  organizations  that  are  eval¬ 
uated  during  an  acquisition  use  of  the  method.  In  an  acquisition  these  are  the  offerors  (or  all 
of  the  offerors  remaining  after  a  competitive  range  determination),  and  possibly  their  subcon¬ 
tractors. 

Development  team  approach:  An  attribute  used  in  SCE.  It  is  related  to  how  the  developer 
organizes  itself  to  produce  the  system;  the  degree  to  which  various  groups  interact  and  are 
brought  to  bear  on  the  effort. 

Directive:  An  order  or  instruction  describing  actions  that  must  be  performed  and  authorizing 
their  performance. 

Document:  Any  lasting  representation  of  information  available  to  the  people  doing  develop¬ 
ment  and  management  work.  A  document  can  be  viewed  as  an  external  memory  for  people. 
Documents  can  be  paper  or  electronic.  Any  process  artifact  can  be  considered  a  “document” 
in  an  SCE. 
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Document  review:  One  of  four  primary  data  collection  sources  used  in  SCE.  The  process  of 
examining  documents  to  find  evidence  of  the  processes  used  by  a  development  organization. 
Documents  can  define  and  standardize  processes,  can  indicate  commitment  to  use  the  pro¬ 
cesses,  can  provide  an  audit  trail  of  processes  that  were  used,  and  can  reflect  data  about  pro¬ 
cess  performance.  Three  levels  of  documents  are  reviewed  during  an  SCE:  organization- 
level,  project-level,  and  implementation-level. 

Environment:  An  attribute  used  in  SCE.  It  refers  to  the  hardware,  software,  and  telecommu¬ 
nications  environment  used  to  develop  the  system. 

Evidence:  Data  on  which  a  judgment  or  conclusion  can  be  based  [AHD  85]. 

Effective  process:  A  process  that  can  be  characterized  as  practiced,  documented,  enforced, 
trained,  measured,  and  capable  of  being  improved  [Paulk  93b]. 

Evolution:  A  gradual  process  in  which  something  changes  into  a  different  and  usually  more 
complex  or  better  form  [AHD  85]. 

Evaluator:  Evaluate,  to  examine  and  judge  carefully.  [AHD  85}.  In  the  context  of  SCE,  evalu¬ 
ator  is  referring  to  the  individual  on  a  team  performing  an  evaluation  on  behalf  of  a  sponsor. 

Fact:  A  statement  whose  content  can  be  verified  as  true  through  the  senses  [Masters  95]. 

Feature:  One  of  a  set  of  process  attributes  that  provide  a  view  of  “whether  the  implementation 
and  institutionalization  of  a  key  practice  are  effective,  repeatable,  and  lasting”  [Paulk  93b].  The 
features  used  in  SCE  come  directly  from  the  common  features  of  CMM  vl  .1 .  They  add  a  level 
of  detail  that  is  appropriate  for  generating  topics  for  investigation.  Examples  of  features  are 
policies,  resources,  and  training.  Features  are  listed  within  each  common  feature  defined  in 
this  glossary.  (See  common  feature.) 

Fidelity:  Faithfulness  to  obligations,  duties,  or  observances.  [AHD  85].  Fidelity  in  an  appraisal 
means  adhering  strictly  to  the  reference  model  used  to  appraise  processes.  CMM  fidelity  re¬ 
fers  to  the  use  of  CMM  components,  and  CMM  components  alone,  as  the  basis  for  rating  an 
organization's  software  process  maturity  [Masters  95].  A  method  shows  good  fidelity  if  it  is 
consistent,  repeatable,  accurate,  and  precise.  Its  results  are  thus  comparable  across  and  with¬ 
in  organizations,  and  errors  are  minimized.  Fidelity  is  closely  related  to  reliability. 

Findings:  Findings  are  the  primary  output  from  executing  the  SCE  method.  Final  findings 
are  used  to  develop  the  findings  briefing  and  final  report.  Findings  are  validated  observations. 
Findings  consist  of  strengths,  weaknesses,  or  improvement  activities  in  one  of  the  reference 
model  components  within  the  scope  of  the  appraisal.  Findings  may  also  be  generated  in  non¬ 
reference  model  areas  from  data  that  does  not  directly  correspond  to  the  reference  model 
used,  but  that  are  significant  to  the  success  of  the  organization’s  operations.  (See  results.) 
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An  observation  that  has  been  accepted  by  the  team  as  valid.  Findings  include  strengths, 
weaknesses,  evidence  of  alternative  practices,  and  evidence  of  non-applicable  practices.  A 
set  of  findings  should  be  accurate,  corroborated,  and  consistent  within  itself  [Masters  95]. 

Goal:  A  summary  of  the  key  practices  of  a  key  process  area  that  can  be  used  to  determine 
whether  an  organization  or  project  has  effectively  implemented  the  key  process  area  [Masters 
95], 

IDEAL  approach:  A  systems  approach  or  life  cycle  framework  for  implementing  process  im¬ 
provement  activities.  IDEAL  stands  for  the  five  phases  of  the  approach:  Initiating,  Diagnosing, 
Establishing,  Acting,  and  Leveraging  [Radice  93], 

Implementation-level  documents:  The  third  of  three  levels  of  documents  reviewed  during 
an  SCE.  These  are  documents  which  provide  an  audit  trail  of  processes  that  were  used,  and 
can  be  used  by  the  development  organization  to  collect  data  about  process  performance. 

Improvement  activity:  A  process  improvement  that  is  not  yet  institutionalized — for  example, 
a  pilot  program  that  implements  a  new  configuration  management  process.  In  SCE,  it  indi¬ 
cates  potential  mitigation  of  risk  due  to  implemented  process.  In  this  sense,  an  improvement 
activity  is  a  weakness  that  if  institutionalized  would  be  considered  a  strength. 

Inconsistency:  An  apparently  contradictory  response  from  the  same  project  to  two  (or  more) 
questions  on  the  questionnaire,  or  from  other  data  collection  mechanisms,  that  relate  to  the 
same  process  area.  May  indicate  an  issue  that  needs  to  be  probed  further.  Related  to  anom¬ 
aly. 

Inference:  A  conclusion  based  on  a  fact.  They  are  not  facts.  In  SCE,  strong  inferences  may 
be  used  as  a  basis  for  observations,  in  addition  to  facts.  Strong  inferences  are  readily  verifi¬ 
able  by  further  data  collection. 

Institutionalization:  The  building  of  infrastructure  and  corporate  culture  that  support  meth¬ 
ods,  practices,  and  procedures  so  that  they  are  the  ongoing  way  of  doing  business,  even  after 
those  who  originally  defined  them  are  gone  [Masters  95]. 

Institutionalization  common  feature:  One  of  four  common  features  in  the  CMM  for  Software 
that  are  related  to  institutionalizing  methods,  practices,  and  procedures:  commitment  to  per¬ 
form,  ability  to  perform,  measurement  and  analysis,  and  verifying  implementation  [Paulk  93b], 

Instrument:  One  of  four  primary  data  collection  sources  used  in  SCE.  An  instrument  is  typi¬ 
cally  a  questionnaire,  survey,  profile,  or  other  written  item  used  to  collect  data.  Instrument  data 
is  typically  collected  and  analyzed  prior  to  the  site  visit. 

Internal  evaluation:  One  SCE  application  type.  Various  internal  evaluation  uses  are  tailored 
applications  of  the  SCE  method.  Typical  internal  evaluation  uses  include:  process  baselining, 
process  improvement  progress  measurement,  process  audits,  and  domain,  product  line,  or 
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project  specific  appraisals.  Preparing  for  an  external,  customer  led  evaluation  is  often  a  rea¬ 
son  that  an  organization  conducts  an  internal  evaluation.  Related  to  acquisition  and  process 
monitoring  SCE  applications. 

Interviewing:  One  of  four  primary  data  collection  sources  used  in  SCE.  The  process  of  ques¬ 
tioning  personnel  from  the  development  organization  to  find  evidence  of  the  processes  used 
by  the  development  organization.  Interviews  provide  insight  into  how  processes  are  imple¬ 
mented  and  show  the  extent  to  which  processes  have  been  internalized  by  members  of  the 
development  organization. 

Judgment:  The  exercise  of  making  sound  and  reasonable  decisions  (verb)  [AHD  85].  In  SCE, 
judgments  refer  to  individual  and  team  decisions  in  the  data  transformation  process  from 
notes  to  observations,  observations  to  findings,  and  findings  to  ratings.  (See  notes,  observa¬ 
tions,  findings,  and  ratings.) 

Key  Practice:  The  infrastructures  and  activities  that  contribute  most  to  the  effective  imple¬ 
mentation  and  institutionalization  of  a  key  process  area  [Paulk  93b]. 

Key  process  area  (KPA):  “A  cluster  of  related  activities  that,  when  performed  collectively, 
achieve  a  set  of  goals  considered  important  for  establishing  process  capability”  [Paulk  93b]. 
Each  KPA  contributes  to  the  environment  in  which  development  organizations  create  software 
products.  Within  the  CMM,  the  KPAs  are  organized  into  five  basic  levels  of  process  maturity 
to  describe  the  progression  from  an  ad  hoc  software  process  to  one  that  is  well  defined  and 
can  act  as  a  stable  foundation  for  continuous  process  improvement. 

Language(s):  An  attribute  for  SCE.  This  attribute  indicates  the  programming  languages  in 
which  the  code  is  to  be  written,  or  in  which  it  has  been  written. 

Mapping:  The  relationship  between  actual  practices  in  the  software  process  implementation 
and  the  process  areas  within  the  reference  model  used. 

Maturity  level:  “A  well-defined  evolutionary  plateau  toward  achieving  a  mature  software  pro¬ 
cess”  [Paulk  93b]. 

Maturity  model:  A  model  of  organizational  activity  used  for  evaluating  a  development  orga¬ 
nization’s  process  capability.  The  maturity  model  has  a  defined  structure,  and  is  available  to 
the  public.The  maturity  model  used  in  SCE  V3.0  is  the  Capability  Maturity  Model  (CMM)  for 
Software  VI  .1  [Paulk  93a]. 

Measurement  and  analysis:  One  of  five  common  features  in  the  CMM  for  Software.  This 
common  feature  describes  the  need  to  measure  the  process  and  analyze  the  measurements. 
Measurement  and  analysis  typically  includes  the  feature  of  examples  of  the  measurements 
that  could  be  taken  to  determine  the  status  and  effectiveness  of  the  Activities  Performed. 
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Method:  A  means  or  manner  of  procedure,  especially  a  regular  and  systematic  way  of  accom¬ 
plishing  something  [AHD  85].  An  appraisal  method  consists  of  appraisal  activities,  processes, 
and  rating  strategies  along  with  associated  data  structures,  definitions,  and  usage  instruc¬ 
tions.  (See  appraisal  method.) 

Method  tailoring:  Making,  altering,  or  adapting  to  a  particular  end  [AHD  85].  In  SCE,  tailoring 
refers  to  selecting  options,  based  on  the  appraisal  goals,  that  may  affect  appraisal  risk.  The 
selection  process,  led  by  the  team  leader  during  appraisal  planning,  of  refining  or  extending 
the  standard,  or  baseline,  method  to  best  fit  the  needs  of  the  sponsor  and  the  appraisal  goals 
defined  during  requirements  analysis.  In  SCE  the  principal  tailoring  options  include  varying  the 
organizational  scope,  reference  model  scope,  and  rating  baseline.  These  options  in  turn  drive 
lower  level  tailoring  options  for  team  size,  skills  and  experience,  and  time  on  site.  There  are 
also  numerous  low  level  implementation  options  relating  to  forms,  templates,  and  instruments 
(appraisal  method  artifacts)  available  for  conducting  the  appraisal. 

Notes:  The  transcription  of  raw  input  data  (from  instruments,  presentations,  interviews,  and 
documents)  by  an  individual  team  member,  usually  in  the  form  of  written  text,  into  information 
formatted  such  that  it  can  later  be  used  to  form  observations  about  processes.  In  SCE,  the 
formatting  is  done  by  various  means,  including  “tagging”  notes  relative  to  the  reference  model 
used. 

Observation:  An  inference  or  judgment  that  is  acquired  from  or  based  on  observing  [AHD  85]. 
An  observation  is  information  extracted  from  the  notes  of  data  collection  sessions  [Masters 
95],  Observations  are  classified  in  terms  of  strengths  and  weaknesses,  and  categorized  by 
reference  model  component.  In  SCE,  observations  are  always  based  on  facts  or  strong  infer¬ 
ences. 

Organization-level  documents:  The  first  (or  top)  level  of  three  levels  of  documents  reviewed 
during  an  SCE.  These  are  the  policies  and  procedures  which  establish  the  development  en¬ 
vironment  for  all  company  project  activities.  Organizational  level  documents  define  the  pro¬ 
cess  and  management  constraints  the  organization  places  on  projects. 

Organizational  scope:  The  part  of  the  appraisal  scope  that  defines  the  breadth  of  the  inves¬ 
tigation  within  the  development  organization.  Typically  described  in  terms  of  a  project  or  num¬ 
ber  of  projects,  but  may  also  relate  to  a  product  line  or  domain.  The  organizational  units  that 
comprise  the  entity  being  appraised  [Masters  95].  (See  appraisal  scope.) 

Outcome:  How  the  findings  (SCE  results)  are  used  by  the  sponsoring  organization — for  ex¬ 
ample,  in  risk  determination  for  an  acquisition,  risk  management  for  process  monitoring,  or 
process  improvement  for  an  internal  evaluation. 

Policy:  “A  guiding  principle,  typically  established  by  senior  management,  adopted  by  an  or¬ 
ganization  to  influence  and  determine  decisions"  [Paulk  93b]. 
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Precedence:  An  attribute  used  in  SCE.  This  attribute  indicates  whether  the  principal  stake¬ 
holders  in  the  system  (acquirer,  end  user,  developer)  have  experience  with  the  type  of  system 
to  be  built.  Systems  that  are  providing  a  new  capability  tend  to  have  more  changes  to  the  re¬ 
quirements  than  do  ones  that  are  replacing  existing  systems. 

Presentations:  One  of  four  primary  data  collection  sources  used  in  SCE.  Presentations  can 
either  be  delivered  by  the  appraisal  team  to  the  recipient  organization,  or  can  be  delivered  by 
the  recipient  organization  to  the  appraisal  team.  Usually  these  presentations  are  provided  in 
a  viewgraph,  briefing  format  allowing  interaction  between  the  team  and  the  participants.  Pre¬ 
sentations  can  delivered  either  for  the  purpose  of  data  collection  or  data  validation.  (See  data 
collection  and  validation.) 

Procedure:  A  written  description  of  a  course  of  action  to  be  taken  to  perform  a  given  task 
[IEEE  91], 

Process:  A  sequence  of  steps  performed  for  a  given  purpose  [IEEE  91]. 

Process  capability:  “The  range  of  expected  results  that  can  be  achieved  by  following  a  pro¬ 
cess”  [Paulk  93b]. 

Process  maturity:  The  extent  to  which  a  specific  process  is  explicitly  defined,  managed, 
measured,  controlled,  and  effective.  Maturity  implies  a  potential  for  growth  in  capability  and 
indicates  both  the  richness  of  an  organization's  software  process  and  consistency  with  which 
it  is  applied  in  projects  throughout  the  organization  [Paulk  93a]. 

Process  monitoring:  One  of  the  primary  applications  of  the  SCE  method.  In  process  moni¬ 
toring,  SCE  results  can  serve  as  an  input  for  an  incentive/award  fee,  as  a  basis  for  value  en¬ 
gineering  incentive  payments,  or  can  be  used  to  help  the  sponsoring  organization  tailor  its 
contract  monitoring  efforts. 

Procuring  Contracting  Officer  (PCO):  The  PCO  is  the  acquisition  agency  person  responsi¬ 
ble  for  all  communications  with  the  offerors  (development  organizations)  in  an  acquisition  ap¬ 
plication  of  SCE.  The  PCO  ensures  that  the  entire  source  selection  process  is  consistent  with 
applicable  regulations.  The  PCO  is  also  responsible  for  advising  the  sponsor  on  the  interpre¬ 
tation  of  the  findings  to  ensure  a  consistent  and  objective  award  decision. 

Product  profile:  See  Profiles. 

Product  type:  An  attribute  in  SCE.  The  product  type  attribute  refers  to  the  particular  aspect 
of  the  application  domain  which  the  system  will  support  or  to  the  type  of  service  which  the  sys¬ 
tem  will  provide.  For  example,  displays  or  communications  could  be  product  types  in  a  com¬ 
mand  and  control  system,  a  weapons  system,  or  another  application  domain.  Although  there 
may  be  similarities  in  the  communications  subsystem  in  the  various  application  domains,  they 
each  have  their  own  set  of  unique  problems  which  must  be  addressed. 
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Profiles:  A  profile  is  the  set  of  attributes  (such  as  Application  Domain,  Product  Type,  and 
Size)  associated  with  a  product  and  the  environment  that  supports  development  of  the  prod¬ 
uct.  There  are  three  types  of  product  profiles  used  in  SCE:  a  “target”  Product  Profile  created 
by  the  sponsor  organization,  representing  the  customer  view  and  reflecting  a  “desired”  state; 
Product  Profile(s)  from  the  recipient  reflecting  attributes  of  a  current  effort(s);  and  a  “proposed” 
Product  Profile  created  by  the  offeror  in  response  to  an  acquisition  application  reflecting  the 
developer  view  of  planned  work. 

Project:  An  undertaking  requiring  concerted  effort,  which  is  focused  on  developing  and/or 
maintaining  a  specific  product.  The  product  may  include  hardware,  software  and  other  com¬ 
ponents.  Typically  a  project  has  its  own  funding,  cost  accounting,  and  delivery  schedule  [Mas¬ 
ters  95]. 

Project-level  documents:  The  second  of  three  levels  of  documents  reviewed  during  an  SCE. 
These  are  documents  which  define  the  development  processes  in  use  for  a  particular  project. 
Project  level  documents  define  the  detailed  processes  that  are  used  to  manage,  coordinate, 
and  integrate  the  engineering  activities  required  for  the  development. 

Rating:  A  position  assigned  on  a  scale;  standing.  [AHD  85]  Ratings  are  judgments  associated 
with  findings.  A  characterization  of  an  organization's  process  relative  to  a  component  of  the 
reference  model  used  in  the  appraisal.  Rating  types  in  SCE  include  satisfied,  not  satisfied,  not 
rated,  or  not  applicable.  The  rating  scale  for  maturity  level  is  taken  directly  from  the  definition 
contained  in  the  reference  model  (e.g.,  Levels  1-5  in  the  CMM  for  Software).  Ratings  can  be 
applied  to  any  component  of  the  reference  model  that  is  planned  for  by  the  team  to  achieve 
appraisal  goals  and  if  collected  data  meets  all  method  rules  for  coverage  and  corroboration. 
(See  appraisal  goals,  coverage  and  corroboration.) 

Rating  baseline:  “Base”  is  the  supporting  part  or  layer;  foundation.  The  fundamental  principle 
or  underlying  concept  of  a  system  or  theory;  basis.  The  fact,  observation,  or  premise  from 
which  a  reasoning  process  is  begun”  [AHD  85].  A  baseline  is  a  specification  or  product  that 
has  been  formally  reviewed  and  agreed  upon,  that  thereafter  serves  as  the  basis  for  further 
development.. .[Paulk  93b],  In  SCE,  choosing  the  rating  baseline  option  is  the  fundamental 
method  tailoring  decision,  made  during  appraisal  requirements  analysis.  This  decision  drives 
subsequent  planning  and  execution  of  the  method.  It  specifies  the  choice  of  method  “rigor” 
made  by  the  sponsor  (in  consultation  with  the  team  leader  or  senior  site  manager).  It  reflects 
the  reference  model  scope  and  coverage  requirements  enabling  team  rating  judgments  to  be 
made.  The  SCE  method  provides  two  rating  baseline  options:  depth  oriented  and  breadth  ori¬ 
ented  (See  Method  Description  for  more  detail.) 

Recipient:  The  appraised  entity  that  receives  the  appraisal.  Synonymous  with  development 
organization.  (See  appraised  entity  and  development  organization.) 
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Reference  model  scope:  The  part  of  the  appraisal  scope  that  defines  the  depth  within  the 
reference  model  used  that  will  be  investigated  during  the  SCE.  Items  outside  the  defined 
scope  of  the  SCE  cannot  be  looked  at  during  an  acquisition  application  of  SCE.  (See  appraisal 
scope.) 

Reliability:  The  ability  of  a  system  or  component  to  perform  its  required  functions  under  stat¬ 
ed  conditions  for  a  specified  period  of  time  [IEEE  90].  In  SCE,  the  method  is  the  “system.”  Re¬ 
liability  is  generally  used  to  refer  to  the  repeatability  and  consistency  of  the  appraisal  method. 
The  ability  to  attain  appraisal  results  that  accurately  characterize  an  organization's  software 
process  [Masters  95], 

Repeatability:The  ability  to  attain  the  same  appraisal  results  if  an  appraisal  of  identical  scope 
is  conducted  more  than  once  in  the  same  time  period  [Masters  95]. 

Request  for  Proposal  (RFP):  A  government  acquisition  document  that  describes  character¬ 
istics  of  the  system  the  sponsor  wants  to  acquire.  It  is  used  in  an  acquisition  application  of  the 
SCE  method.  This  document  is  used  to  solicit  proposals  from  commercial  development  orga¬ 
nizations  (offerors)  and  to  communicate  the  characteristics  of  the  desired  system  to  the  offer¬ 
ors.  In  source  selection,  this  is  the  document  that  specifies  that  an  SCE  will  be  performed,  how 
it  will  be  performed,  and  what  is  expected  of  the  offerors  to  respond  to  the  customer’s  need. 
The  RFP  is  a  key  artifact  describing  the  appraisal  plan  in  an  acquisition  application. 

Results:  A  synonym  for  the  SCE  findings.  This  is  the  primary  output  of  the  SCE.  In  addition 
to  findings,  the  results  may  include  reference  model  ratings  (such  as  maturity  level),  and  draft 
recommendations,  depending  on  the  application  of  the  method  and  the  goals  documented  in 
the  appraisal  plan.  Appraisal  results  are  always  provided  to  the  sponsor,  and  should  be  pro¬ 
vided  to  the  appraisal  recipient  (development  organization).  (See  findings,  appraisal  outputs.) 

Reuse  estimate:  An  attribute  used  in  SCE.  It  indicates  the  development  organization’s  ap¬ 
proach  to  building  the  product.  It  is  correlated  to  the  size  attribute. 

Sampling:  A  set  of  elements  drawn  from  and  analyzed  to  estimate  the  characteristics  of  a 
population.  During  an  appraisal,  data  collection  is  planned  to  provide  a  sampling  of  the  pro¬ 
cess  data  related  to  the  reference  model  components,  organizational  units,  and  life  cycle 
phases  within  the  scope  of  the  appraisal  [Masters  95]. 

Site:  A  geographic  location  of  one  or  more  of  an  organization's  units  that  participate  in  an  ap¬ 
praisal. 

Site  information  packet:  The  set  of  materials  requested  by  the  sponsor,  and  provided  to  the 
appraisal  team  by  the  recipient  organization,  for  use  in  planning  and  preparing  for  the  apprais¬ 
al.  In  SCE,  it  includes  information  such  as  organization  charts,  site  terminology  lists,  document 
hierarchy  and  model  content  mapping,  product  profiles  (including  the  proposed  product  profile 
for  an  acquisition  SCE),  responses  to  instruments,  etc. 
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Site  technical  coordinator:  The  technical  focal  point  assigned  by  the  recipient  organization 
to  assist  and  facilitate  the  appraisal  team  in  conducting  its  activities. 

Site  visit:  The  collection  of  SCE  activities  that  encompass  the  investigation  by  the  SCE  team 
at  a  development  organization’s  site. 

Size:  An  attribute  for  SCE.  The  size  attribute  indicates  the  magnitude  of  the  product  (and 
hence  the  required  project).  Size  is  composed  of  three  related  attributes.  The  contract  duration 
is  the  estimated  or  required  length  of  time  for  the  development  of  the  product.  The  team  size 
is  the  number  of  developers  who  will  be  involved  in  the  project.  The  estimated  size  is  the 
amount  of  code  to  be  developed  (in  a  software  system). 

Software  Capability  Evaluation  (SCE):  A  method  for  evaluating  the  software  process  of  an 
organization  to  gain  insight  into  its  software  development  capability.  SCE  can  also  be  defined 
as  a  method  for  evaluating  the  processes  of  an  organization  to  gain  insight  into  its  business 
capability.  Which  model  processes  are  evaluated  is  determined  by  the  sponsor  during  ap¬ 
praisal  planning  (e.g.,  software,  people,  acquisition). 

Software  development  plan  (SDP):  “The  collection  of  plans  that  describe  the  activities  to  be 
performed  for  the  software  project”  [Paulk  93b]. 

Software  process  capability:  “The  range  of  expected  results  that  can  be  achieved  by  follow¬ 
ing  a  process”  [Paulk  93b].  For  purposes  of  an  SCE,  software  process  capability  reflects  those 
processes  which  provide  an  environment  for  development  teams  to  produce  software  prod¬ 
ucts.  The  processes  evaluated  include  decision  making  processes  (such  as  software  project 
planning),  communication  processes  (such  as  intergroup  coordination)  and  technical  process¬ 
es  (such  as  peer  reviews).  (See  process  capability  and  process  maturity.) 

Software  process  implementation:  A  tailored  set  of  practices  that  defines  how  software  de¬ 
velopment  work  is  supposed  to  be  done. 

Source  selection:  The  government  term  for  a  acquisition  process  to  select  a  supplier.  An  ac¬ 
quisition  application  of  the  SCE  method  is  used  to  provide  results  that  are  factored  into  the 
source  selection  decision.  In  source  selection,  the  results  of  the  SCE  are  used  by  the  spon¬ 
soring  organization  to  characterize  the  process-related  risk  of  awarding  a  contract  to  an  offer¬ 
or.  SCE  is  only  one  criterion  among  many  used  to  select  contractors  in  government 
acquisitions.  (See  acquisition  and  acquisition  agency.) 

Source  Selection  Authority  (SSA):  The  individual  responsible  for  the  conduct  of  the  govern¬ 
ment  source  selection  (acquisition)  process,  in  an  acquisition  application  of  the  SCE  method, 
the  SSA  is  the  sponsor.  The  SSA  is  the  final  arbiter  on  the  use  of  SCE,  approves  how  the  SCE 
results  will  influence  the  award  decision,  and  makes  the  award  decision.  (See  acquisition,  ac¬ 
quisition  agency,  source  selection  advisory  council,  and  source  selection  advisory  board.) 
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Source  Selection  Advisory  Council  (SSAC):  The  SSAC  is  chartered  by  the  sponsoring  or¬ 
ganization  (acquisition  agency)  with  collecting  and  analyzing  the  evaluations  of  each  offeror. 
This  group  performs  risk  assessment  activities.  This  is  the  only  group  permitted  to  compare 
the  SCE  results  (strengths  and  weaknesses)  of  the  offerors  against  one  another.  The  SSAC 
may  recommend  to  the  sponsor  how  the  SCE  findings  will  be  incorporated  into  the  award  de¬ 
cision  at  the  pre-RFP  release  briefing. 

Source  Selection  Evaluation  Board  (SSEB):  This  is  the  government  group  that  evaluates 
the  offerors’  proposals  against  defined  evaluation  standards  in  an  acquisition  application  of 
SCE.  This  group  performs  risk  identification  tasks.  This  group  develops  the  evaluation  stan¬ 
dards  and  receives  approval  to  use  them  from  sponsor  before  the  issuance  of  the  RFP.  The 
SSEB  is  usually  organized  into  technical  and  cost  teams  important  to  the  award  decision.  If 
the  findings  of  an  SCE  are  being  factored  into  the  source  selection  decision  as  an  Evaluation 
Criterion,  the  SCE  team  leader  should  be  a  member  of  the  SSEB.  The  SSEB  prepares,  prior 
to  the  release  of  the  RFP,  an  evaluation  standard  that  will  incorporate  the  SCE  results  into  the 
source  selection  process. 

SSEB  Chairperson:  The  SSEB  chairperson  coordinates  all  activities  of  the  SSEB  related  to 
the  acquisition.  The  chairperson  will  facilitate  the  incorporation  of  SCE  into  the  source  selec¬ 
tion  documentation  and  monitor  the  various  evaluation  teams,  including  the  SCE  team. 

Sponsor:  The  decision  maker  in  the  organization  that  commissions  the  SCE  to  be  performed 
and  uses  the  findings  (results).  Evaluator  results  are  always  provided  to  the  sponsor.  The  in¬ 
dividual  who  authorizes  an  evaluation,  defines  its  goals  and  constraints,  and  commits  to  use 
of  evaluation  outputs  [Masters  95]. 

Sponsoring  organization:  The  organization  that  commissions  the  SCE  to  be  performed  and 
uses  the  findings.  (See  sponsor  and  acquisition  agency.) 

Standard:  “Mandatory  requirements  employed  and  enforced  to  prescribe  a  disciplined,  uni¬ 
form  approach  to  software  development”  [Paulk  93b]. 

Strength:  Indicates  the  team  judgment  of  an  effective  implementation  of  a  component  of  the 
reference  model.  In  SCE,  a  strength  further  indicates  a  particular  part  of  the  organization’s  ca¬ 
pability  that  is  sufficiently  robust  to  mitigate  the  development  risks  due  to  process. 

Implementation  of  practices  which  in  an  appraisal  team's  judgment,  improve  an  organization's 
software  process  capability.  CMM  related  strengths  are  effective  implementation  of  one  or 
more  of  the  CMM  key  practices  or  one  or  more  alternative  practices  that  contribute  equivalent¬ 
ly  to  the  satisfaction  of  KPA  goals  [Masters  95]. 

Subcontractor:  A  development  organization  that  is  contracted  to  work  for  another  develop¬ 
ment  organization  to  produce  products. 
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Subcontractors:  An  attribute  in  SCE.  This  attribute  is  used  to  indicate  whether  the  develop¬ 
ment  organization  intends  to  use  subcontractors  in  the  development,  and  is  a  factor  if  they  lack 
experience  with  subcontract  management. 

Sufficiency  for  rating:  The  extent  to  which  observations  meet  appraisal  method's  rules,  thus 
satisfying  the  prerequisites  for  rating.  Sufficiency  judgments  are  composed  of  a  series  of  team 
judgments  regarding  the  validation,  coverage,  and  corroboration  aspects  of  observations. 

Target:  An  attribute  in  SCE.  This  attribute  indicates  the  hardware  configuration  that  the  devel¬ 
oped  software  will  run  on  when  operational. 

Target  Process  Capability:  The  process  capability  that  is  most  appropriate  for  the  planned 
development;  the  process  capability  desired  by  the  sponsoring  organization  for  the  product  to 
be  developed.  The  Target  Process  capability  consists  of  a  set  of  process  areas  within  the  ref¬ 
erence  model  used,  and  helps  establish  the  boundaries  of  the  SCE  investigation — a  process 
area  is  evaluated  if  and  only  if  it  is  part  of  the  Target  Process  Capability. 

Topic:  A  topic  is  a  focused  subject  matter  area  probed  during  the  SCE  investigation.  Topics 
are  a  subset  of  process  activities  that  work  towards  achieving  a  specific  process  area  goal. 
Topics  are  intended  to  be  detailed  enough  to  focus  the  investigation  on  observable,  docu¬ 
mented  work  practices,  but  sufficiently  abstract  that  they  avoid  prescribing  how  the  process 
area  is  implemented.  Topics  are  selected  by  considering  the  intersection  of  a  process  area 
goal  and  its  associated  reference  model  features. 

Type  of  Work:  An  attribute  for  SCE.  This  attribute  indicates  the  portion  of  the  development 
life  cycle  which  will  be  performed.  As  examples  of  different  types  of  work,  in  “full  software  de¬ 
velopment”  a  development  organization  is  required  to  build  a  product  based  on  the  system  re¬ 
quirements,  while  in  “code  development  only”  the  development  organization  is  required  to 
develop  code  according  to  the  system  requirements  and  software  top  level  design  provided 
by  the  issuing  authority. 

Use  of  the  SCE  method:  Executing  the  SCE  method  within  a  particular  context.  The  principal 
high-level  uses  of  the  SCE  method  are  in  acquisition,  and  process  monitoring,  and  internal 
evaluation.  This  is  sometimes  referred  to  as  the  application  of  the  method. 

Validation:  To  substantiate;  verify.  Valid  refers  to  producing  the  desired  results  [AHD  85].  In 
SCE,  validation  refers  to  the  process  of  substantiating  observations  made  about  processes, 
using  rules  for  confirming  observations  defined  in  the  method.  A  valid  observation  is  one  that 
is  accurate  and  has  been  agreed  to  by  the  team  through  a  consensus  process.  Validated  ob¬ 
servations  are  equivalent  to  findings  after  the  team  concludes  that  data  coverage  and  corrob¬ 
oration  rules  have  been  met.  The  rationale  for  validation  is  related  to  the  data  element 
objective  of  an  SCE,  obtaining  an  accurate  picture  of  process  capability  at  a  site. 
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Verifying  implementation:  One  of  five  common  features  in  the  CMM  for  Software.  This  com¬ 
mon  feature  describes  the  steps  to  ensure  that  the  activities  are  performed  in  compliance  with 
the  process  that  has  been  established.  Verifying  implementation  typically  encompasses  the 
features  of  reviews  and  audits  by  management,  quality  assurance,  and  other  support  units. 

Weakness:  Indicates  the  team  judgment  that  an  effective  implementation  of  a  component  of 
the  reference  model  is  not  institutionalized.  In  SCE,  a  weakness  indicates  a  particular  part  of 
the  organization’s  capability  that  has  characteristics  that  increase  the  risks  due  to  process. 

Ineffective  implementation  of  or  lack  of  practices  which,  in  an  appraisal  team's  judgment,  in¬ 
terfere  with  effective  performance  of  software  development  tasks.  CMM  related  weaknesses 
are  an  ineffective  implementation  or  lack  of  implementation  of  one  or  more  CMM  key  practices 
with  no  acceptable  alternative  practices  in  place  [Masters  95]. 
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